AI is not good at information retrieval

A lot of people who are new or less experienced with GenAI complain about the way it makes stuff up (known as hallucination). Sometimes this comes down to a misunderstanding of how these systems work with information.

For most of our internet-enabled life, we’ve used search engines that look for matching text and keywords on pre-existing webpages. In other words, we’ve used it for information retrieval.

GenAI can be quite unreliable in certain situations. For instance, I once asked it for the current date, and it repeatedly provided the incorrect date, even after I pointed out the error. I mean, even the most ancient Candybar Nokia phone can sync to a satellite to tell me what time it is. How is that this supposedly life-changing technology can’t even tell me what day it is?!

The reason is that these models are generative, meaning they produce text, images, etc on the fly, after being “trained” on many many documents and texts. It isn’t “looking things up.” It’s using its pattern recognition algorithms to generate new text (or pictures or music or whatever).

While LLMs are pretty magical, information retrieval just isn’t a strong suit in many cases. That’s not to say it never works; I’ve had success learning about certain topics with an LLM. It’s just not as reliable as say, Google.

Note: it now gives me the right date and time if I ask it, but not before “analyzing” which involves communicating with some external source to determine the time. Here’s how GPT describes it:

“When I perform tasks like checking the current time, I use a combination of my generative capabilities and external tools or databases to ensure accuracy. In this case, I used a Python script to get the precise current time, rather than generating a response based solely on my training data. This ensures that the information is accurate and up-to-date.”

In other words, it does what you could do by looking at your phone and checking the time. This is important when you consider when and how to use these sorts of tools.