Why search fails

A feature that answers questions from your own manuals, tickets or contracts has two halves. The first half finds the passages that might hold the answer. The second half, the language model, writes an answer from those passages. This pattern is often called RAG, short for retrieval-augmented generation.

When such a feature gives a wrong answer, people usually blame the model and try a bigger one. Most of the time that changes nothing, because the model only sees what retrieval hands it. If the right passage is not among them, no model can write the right answer. It can only write a confident wrong one.

Here are the five failures we look for first, and what fixes each.

The right passage was never found

This is the most common failure, and it usually starts when documents are cut into pieces for the index. A table gets separated from the heading that says what it is. A contract clause gets separated from the definitions it depends on. A step in a procedure ends up in one piece and its warning in the next.

The fix is to cut documents along their own structure instead of every 500 words: by section, by clause, by question in a FAQ. Each piece keeps the path of headings above it, so a passage about “Limits” still knows it belongs to “Travel policy, Hotels”. For tables, we keep the header row with every part of the table.

The second cause is relying on vector search alone. Vector search finds passages that mean something similar, which is excellent for “how do I get my money back” matching a section called “Refunds”. It is weak on exact strings: product codes, error numbers, article numbers, people’s names. Keyword search is the opposite. We use both together, which is called hybrid search, and then re-rank the combined results with a model that reads the question and each passage side by side.

The passage was found, but it was old

Many companies have three versions of the same policy in their shared drive. If all three are in the index, the feature will sometimes quote the one from four years ago, word for word and with a citation.

The fix is not in the model. The index has to follow one source of truth. Each document carries its version and date. A document that is replaced or deleted at the source leaves the index too. Ingestion runs as a scheduled job with monitoring, so a failed sync shows up as an alert rather than as a slow drift into stale answers. Where two current documents disagree, the feature should say so instead of choosing one.

The passage was found, but this user may not see it

An index built from the whole shared drive contains the salary table and the board minutes. A search feature that ignores permissions will happily quote them to anyone who asks the right question.

Permissions belong in retrieval, not in the answer. Every piece in the index carries the access rules of its source document, copied from the system it came from. The search runs with the identity of the person who asks and only ever returns pieces that person may open. Filtering the answer afterwards is too late, because the model has already read the passage.

The answer went beyond the passage

Sometimes retrieval does its job and the model still adds a detail that is not in the source, because it sounds likely. This is what people mean by made-up answers.

Three things reduce it. The prompt tells the model to answer only from the passages and to cite one for each claim. A check after the answer compares each sentence with the passage it cites and flags any sentence without support. And the feature is allowed to say “I could not find this in the documents”. That last one sounds like a weakness. It is the most trusted behaviour a search feature can have, and we measure how often it happens so it does not become an excuse.

The question was not a search query

People ask follow-up questions. “And for contractors?” means nothing on its own. If the feature searches for “and for contractors”, it will find nothing useful.

The fix is a step before search that rewrites the question into a complete query using the conversation so far: “holiday carry-over rules for contractors”. The rewritten query goes into the trace, so when an answer is wrong you can see whether the rewrite was the problem.

How to tell which failure you have

You cannot fix retrieval if you only look at final answers. For every answer, log the query that was actually searched, the passages that came back with their scores, and the passages the model cited. Our note Log every request lists the full set of fields.

Then build a small retrieval test set: for each question, which document or section should be found. Measure that separately from answer quality. If the right passage is in the top five results 60 percent of the time, work on retrieval. If it is there almost always and the answers are still wrong, work on the prompt and the checks. This one split saves weeks of guessing.

A short checklist

• Documents are cut along their own structure, and each piece keeps its headings.
• Search is hybrid: keywords and meaning, then a re-ranking step.
• The index follows one source of truth, with versions, dates and deletions.
• Permissions are applied during search, with the identity of the person asking.
• Every claim in an answer has a citation, and a check verifies it.
• The feature may say it did not find the answer, and you know how often it does.
• Follow-up questions are rewritten before search, and the rewrite is logged.
• Retrieval has its own test set, measured apart from the answers.

None of this is exotic. It is the unglamorous half of the work, and it is where most of the quality of a document-search feature comes from. If yours gives answers you cannot trust, send us a few examples. We will look at the traces with you and tell you which of the five it is.