Why AI Sounds Right and Still Gets Things Wrong

A student asks an assistant for "three research papers on sleep and exam performance, with authors and links". Within seconds comes a neat list: titles that sound right, authors with believable names, journals, years. The student pastes two into their reference list. The third gets a link check later, and the link goes nowhere. On a closer search, none of the three papers exists.

This is not a story about a bad tool. It is a story about a normal behaviour of language models, and about a person who did not know to expect it. Once you understand why it happens, you stop being surprised and start being careful in the right places.

• What counts as a hallucination, including less obvious kinds.
• Why models produce them, in plain words.
• Where errors are most likely, and a list of warning signs.
• Habits that reduce errors, and the limits of those habits.
• A task-fit table to help you decide when to rely on AI and when to check hard.

A hallucination is output that reads as confident and fluent but is false, unsupported by the sources it should be based on, or invented. "Hallucination" is a slightly unfortunate word, because it suggests a rare breakdown. Many people prefer plainer descriptions such as "making things up" or "fabrication", and some research uses the broader word "confabulation". The label matters less than recognising the shapes it takes:

As article 3 explained, a language model chooses a likely next piece of text, over and over. A realistic-looking citation is a very likely continuation of "here are three papers on sleep and exams". The model has seen thousands of citation formats, so it can produce the shape of one fluently. Whether the specific paper exists is a different question, and nothing in plausibility-seeking requires the answer to be yes.

A 2025 research paper by Kalai, Nachum, Vempala and Zhang, "Why Language Models Hallucinate", argues that hallucinations arise partly because training and evaluation procedures reward guessing over acknowledging uncertainty. Think of an exam with no penalty for wrong answers: a student who always guesses scores higher than a student who leaves blanks. If models are scored and tuned in a similar way, they learn that a confident guess beats "I'm not sure". The authors treat this as a feature of how systems are trained and measured, which suggests it can be improved by changing the incentives, but it does not mean any given model today has solved it.

A model sees famous facts many times in training and obscure facts rarely or never. For something it has seen only a handful of times, or not at all, there may be nothing reliable to draw on, and yet the machinery still produces an answer.

If you have not given the model the document it needs, it may answer from general patterns. If you gave it a long document, parts may be overlooked. If your question contains an error or an assumption, the model often continues in that direction.

Models are tuned to produce answers people rate highly, and people often rate agreeable, confident and complete answers highly. The side effect can be sycophancy: agreeing with a mistaken user, softening corrections, or not saying "I don't know".

Because each piece builds on the last, an early mistake can drag a long answer in the wrong direction. Longer answers give more opportunities for a slip.

Learn this table; it tells you where to spend your checking effort.

Notice that the lower-risk column is not "safe". It is "easier to check". The common thread is whether you can quickly tell if the output is wrong.

None of these proves an error, but each justifies a closer look:

• Very specific details with no source: exact figures, dates, page numbers, quotes.
• Sources you cannot find with a quick search of the title or author.
• Names or terms you do not recognise in an area you thought you knew.
• Perfectly round or too-neat numbers.
• No uncertainty anywhere, especially on questions where experts disagree.
• The answer repeats your own assumption back to you as if confirmed.
• A response that fits your hope too perfectly, such as the exact reference you were wishing existed.

Official guidance from the model makers is consistent on this. Anthropic's documentation on reducing hallucinations suggests allowing the model to say "I don't know", asking it to extract word-for-word quotes from long documents before answering, and having it support each claim with a quote and retract claims it cannot support. It also suggests comparing several runs and restricting the model to the supplied documents instead of its general knowledge. And, importantly, it says these techniques reduce hallucinations but do not eliminate them, so critical information should still be validated.

Turning that into habits:

And what does not work reliably:

• "Do not hallucinate" as the only instruction. The model cannot reliably tell which of its outputs are invented. Instructions help most when they change the task, such as "use only this text".
• Asking "Are you sure?" Some assistants will reverse a correct answer, and some will defend a wrong one. It is not a verification method.
• Trusting a longer, more detailed answer. Detail is cheap for a language model.
• Assuming a paid or newer product is immune. Better tools often err less. Not never.

Goal. You need real sources on sleep and exam performance for an assignment.

Initial request.

Give me five academic papers on sleep and exam performance with authors, year and DOI.

What may come back (an illustrative example, not real output):

Example response (illustrative)1. Chen, L. and Rao, P. (2019). Sleep duration and academic outcomes in university students. Journal of Educational Health, 12(3), 45-58. doi:10.0000/example2. ...

What is wrong. Every element looks right, which is the problem. The request asked the model to recall specifics from memory, which is exactly where invented details appear. Any of these papers may be real, partly real, or entirely made up, and you cannot tell from the text.

Better approach. Change the task from recall to search guidance, and move the verification to a real database:

I am writing a short literature note on sleep and exam performance inuniversity students. Do not give me any citations from memory. Instead:1. List 6 search phrases I could use in Google Scholar or my library database.2. Name 3 kinds of study design I should look for and what each can and   cannot show.3. Tell me what a trustworthy source looks like for this topic.

Verify. You run the searches yourself, open the actual papers, and note the citation details from the paper or database record, not from the assistant. If you later paste a real abstract into the assistant, you can ask it to summarise that text, and then check the summary against the abstract.

Result. You used the assistant for the part it handles well (search strategy and explanation), and avoided the part where it is unreliable (recalling bibliographic details).

This is the "Decide" step of the Work Loop in table form.

• Assuming that hallucinations only happen on difficult questions. They also happen on simple-looking ones, such as "who wrote this paper".
• Verifying one item and assuming the rest are fine. One correct citation tells you little about the others.
• Using AI to verify AI. A second assistant can help flag issues, but it can repeat the same error. Your final check should rely on independent sources.
• Believing a source exists because a link is present. Links can be invented or may not support the claim. Open them.
• Feeling safe because the answer is "cautious". Hedged language can be just as wrong as confident language.

Below is the text of our 2023 annual report. Answer only from this text.Question: what does it say about employee attrition?For each statement in your answer, quote the sentence from the report thatsupports it. If the report does not mention attrition, say "not stated".[paste the report text]

• Next: Choosing the Right Kind of AI: Chat, Search, Notebooks and Assistants shows which kinds of tools reduce which risks.
• To act on this right away, read Give AI the Right Context: Source Material and Long Documents.
• For the full checking routine, see How to Fact-Check AI-Generated Answers.