Why does AI "hallucinate"?

I've been thinking lately, why we say the things we say with AI. Why when we get a bad result, we're supposed to call it "hallucination"? Why when it gets something right, or close to right, it's a genius AI model, but when it gets it wrong it's a glitch.

It isn't. See hallucination is a mistake in the human perception system - you see, or hear something that doesn't come from the senses that usually provide you with that information. But with AI, that isn't that. The system works like in any other query, like with any other prompt, it just gives you the wrong answer. You pulled the handle of the great slop machine, and instead of getting 7 7 7, you got something a person would pull out of their ass, with confidence of an expert. The only difference is that you can verifiably say that the answer is incorrect, the rest of the system worked the way that it was intended.

The term was used since the 1980s, but it started to subtly change its meaning around the big strides around AI, to be finally updated in 2023 in the Cambridge Dictionary, to include what we now call AI hallucinations. It's much nicer term compares do something like "error" or "mistake" or "misinformation". It feels like an oopsie, a glitch that can be ironed out, a slip of a tongue in otherwise well functioning system.

Mary Shaw, a professor at Carnegie Mellon University, and a recipient of National Medal of Technology and Innovation put it simply:

The current fashion for calling generative AI's errors 'hallucinations' is appalling. It anthropomorphizes the software, and it spins actual errors as somehow being idiosyncratic quirks of the system even when they're objectively incorrect.

She's not the only one.

It's largely a marketing term, that deflects the responsibility for erroneous, incorrect and just plainly crazy responses. If you had a machine that had some chance of giving you a number between 0 and 100 each time you pressed a button, then you'd say an equation out loud, pressed said button and got a wrong answer, you'd be insane to say that the random number generation "hallucinated". But somehow with AI that's not only ok, it's encouraged to use that term.

A vague, nonjudgmental description of a bad result. All that without a specific definition of when it applies, how incorrect does the answer have to be to fall under that category. Is it only the number of Rs in Strawberry? Is it when AI tells you that a bowling ball will bounce higher than a bouncing ball, and then gives you a scientific-sounding mumbo jumbo? When ChatGPT pretends to hear something that didn't happen and makes stuff up on the spot? Every time you get the never-ending "oh, you're right, my mistake", "you're correct to push back on that", etc.? For those who put and continue to put insane amounts of money into the bubble, it's better if we wouldn't think about it too much. People make mistakes too (and consume water!), so it's literally the same thing. You wouldn't be super mad at your friend for getting a fact wrong here and there, would you, so why scrutinize little-old chatbot. He's doing his darnest best, and it'll get better very very soon (although it's also PhD-level already).