I hate the term "hallucinations" for when AIs say false things. It's perfectly calculated to mislead the reader - to make them think AIs are crazy, or maybe just have incomprehensible failure modes.

AIs say false things for the same reason you do.

At least, I did. In school, I would take multiple choice tests. When I didn't know the answer to a question, I would guess. Schoolchild urban legend said that "C" was the best bet, so I would fill in bubble C. It was fine. Probably got a couple extra points that way, maybe raised my GPA by 0.1 over the counterfactual.

Some kids never guessed. They thought it was dishonest. I had trouble understanding them, but when I think back on it, I had limits too. I would guess on multiple choice questions, but never the short answer section. "Who invented the cotton gin?" For any "who invented" question in US History, there's a 10% chance it's Thomas Edison. Still, I never put down his name. "Who negotiated the purchase of southern Arizona from Mexico?" The most common name in the United States has long been "John Smith", applying to 1/10,000 individuals. An 0.01% chance of getting a question right is better than zero, right? If I'd guessed "John Smith" for every short answer question I didn't know, I might have gotten ~1 extra point in my school career, with no downside.

You can go further.

https://www.astralcodexten.com/p/shameless-guesses-not-hallucinations