When should this make use of LLMs?

[jnothman]

Something that comes to mind when looking a library like this is whether it should be able to make use of LLMs to validate tricky cases, e.g. review low-confidence matches for fuzzy_joiner. Is this a capability you would think about integrating?

[GaelVaroquaux]

Hi @jnothman , nice to see you here.

With regards to large language models, I worry about the operational complexity. Some of these things require expensive resources to run, while skrub is super light. We started studying the benefit of language models: https://arxiv.org/abs/2312.09634 and they do bring benefits in certain cases.

I think that in some future we will add an optional dependency to something huggingface to be able to easily integrate them. IMHO, I don't see them becoming a mandatory dependency any time soon: right now it is so pleasant to use the combination of skrub + sklearn as everything is lightweight and good enough for many projects.

What do you think?

[jnothman]

I'm just trying to explore the space of how we get the best of different methods in a world that is no longer bound by what we can code by hand. If there are cases where LLM solutions have the potential to complement heuristic methods, then I suppose we should consider how to integrate them in ways that make them optional and pluggable, or what would be required to do so.

It does appear that there are opportunities in data set triage and cleaning using LLMs. If that is the case, should we be helping users adopt those technologies with moderation and reliability? What does that look like for OSS libraries?