Hallucination

What it is

A hallucination is output from a language model that reads as confident and well-formed but is not supported — either by the material the model was given, or, where none was given, by verifiable real-world fact. The survey literature on natural language generation split the first case in two: output that contradicts the provided source, and output that cannot be verified from it. Both are fluent, and neither is marked as uncertain in the text itself. Later surveys of large language models widened the term to cover factual error against the real world, because these models typically function as open-ended systems where the failure shows up predominantly as factual error. That widening is itself contested: later work rejects it as conflating hallucination with factuality, and defines hallucination only against the training data and the input context. Definitions are still not uniform across the field, so a claim that a system does not hallucinate means little without the definition behind it.

Why it matters

A model produces text whether or not evidence exists for it. Asked a question beyond its knowledge boundary it is more likely to fabricate an answer than to reject the question: standard fine-tuning forces a model to complete every response rather than let it express uncertainty. Why the behaviour persists after that is argued rather than established — the leading account is that training and evaluation reward guessing over acknowledging uncertainty, because benchmarks graded right-or-wrong score an abstention the same as a wrong answer. That is a tendency, not a law — abstention can be trained, and instruction-tuned models do decline some questions, but the shift from answering to abstaining is selective and incomplete, so a decline cannot be assumed. Adding retrieval does not remove the problem either: retrieval-augmented systems still hallucinate, and the failure can originate in the retrieval stage as well as in generation. This matters for company claims because a model asked about your pricing or your certifications will answer either way, and the answer will read exactly as authoritative as a correct one. Grounding and retrieval boundaries reduce the rate; a maintained record of what is actually true is what makes the reduction measurable rather than assumed. Maintaining that record is the part of the problem Knowledge Company works on.