Why Language Models Hallucinate and Cannot Say "I Don't Know"
Language models hallucinate because their training and evaluation reward guessing over admitting uncertainty. Models are unable to say “I don’t Know” because they focus on accuracy. Guessing can improve scores but leads to more confident errors (hallucinations).
Some facts are unpredictable or unavailable, making errors inevitable. The solution might be roughly penalizing errors more than uncertainty and rewarding honest abstention.
Federico Ulfo