Test before launch

A demo works on the five questions its builder thought of. Users will ask the five thousand questions nobody thought of. Testing is how you find out which of those break before your users find out for you.

Testing an LLM feature is not the same as testing ordinary code. The same input can give a different answer tomorrow. A correct answer can be written in a hundred ways. And the model under the feature can change when the provider ships a new version. None of that makes testing impossible. It changes what a test looks like.

This note describes how we test a feature before launch. It is the same method whether the feature is a chat assistant, a document search or an agent.

Start from real cases, not invented ones

The first job is a test set: a list of inputs, each with a description of what a good result looks like. The quality of everything after this depends on where those inputs come from.

Invented questions are too clean. People who write test questions write them the way they would like users to ask. Real users write with typos, ask two things at once, leave out the context, paste half an email, or ask something the feature was never meant to answer.

So we collect inputs from the places where the real questions already are:

• support tickets and chat logs, for an assistant or a search feature;
• search logs from the current site or intranet, including the searches that returned nothing;
• the documents or forms an agent will process, taken from a normal week of work;
• the people who do the job today, when the product is new and there is no history yet.

Fifty to a hundred cases are enough to start. We make sure the set includes the uncomfortable ones: questions with no answer in the data, questions about another customer’s data, attempts to make the feature say something it should not, and inputs in a second language if your users write in one.

Write down what “good” means for each case

An expected answer written as one perfect string is almost useless, because the model will never produce exactly that string. Instead, each case gets a short description of what a good answer must contain, what it must not contain, and where the information should come from.

{
  "id": "holiday-carry-over-03",
  "input": "can i take my unused days into next year??",
  "must_include": ["up to five days", "expire on 31 March"],
  "must_cite": ["handbook/4.2-holiday"],
  "must_not": ["any promise of payment for unused days"],
  "tags": ["hr", "typo", "short-question"]
}

The tags matter later. When a change breaks something, they tell you whether it broke one kind of question or everything.

Writing these descriptions is the slowest part of the work, and it is the part that needs your people. They know what a correct answer is. We know how to turn that knowledge into checks a machine can run.

Three kinds of graders

A grader is whatever decides whether a result passes. We use three kinds, and most features need all three.

Checks in code

Anything that can be checked with ordinary code should be: the output is valid JSON, every claim has a citation, the cited document exists, the number in the answer matches the number in the source, the answer is shorter than the limit, no email address appears in it. These checks are fast, cheap and never change their mind.

Model graders with narrow questions

Some things need judgement, such as whether the answer actually says that unused days expire in March. For those we ask a second model one narrow question at a time, with a yes or no answer. “Is this answer good?” is a bad grader question. “Does the answer state that carried-over days expire on 31 March?” is a good one.

Model graders make mistakes too, so we check them. We label a sample of results by hand and measure how often the grader agrees with the people. Where it disagrees often, we rewrite the question or move that check to human review.

People, on a sample

Some qualities only a person can judge, such as tone, or whether an answer would confuse a new employee. We do not ask people to review every result on every run. We ask them to review a sample, and every case where the automated graders disagree with each other.

Run the set on every change

In an LLM feature, the prompt is code. So are the model version, the temperature, the retrieval settings, the size of document chunks and the list of tools an agent may use. Any of them can make the feature better on one kind of question and worse on another.

So the test set runs automatically whenever one of them changes, the same way unit tests run on every commit. The run compares the results with the last accepted run, not only with a fixed pass mark.

# runs on every pull request that touches prompts/, retrieval/ or models.yaml
eval:
  dataset: evals/handbook.jsonl
  compare_with: last-accepted
  block_merge_if:
    - pass_rate_drops_by: 2%      # across the whole set
    - any_case_tagged: critical   # fails, even once
    - p95_latency_above: 4s
    - cost_per_answer_rises_by: 25%

The numbers in that file are examples. The right limits depend on the feature, and we agree them with you in the first weeks.

An example. The cases and the numbers are made up for this note.

Read the failures, not only the score

A pass rate of 92 percent sounds good. It may also mean the feature fails every question about refunds, because refunds are eight percent of the set. An average hides exactly the kind of failure that turns into a complaint.

After every run we read the failed cases and group them: retrieval found the wrong document, the answer added something the source did not say, the format was wrong, the question was out of scope and the feature answered anyway. Each group points to a different fix. Rewriting the prompt fixes some groups and does nothing for others.

Speed and cost are results too

Every test run also records how long each answer took and what it cost. A prompt change that improves quality by one point and doubles the cost per answer is a decision for you to make, not something to discover on next month’s invoice. We report the slowest answers as well as the average, because users notice the slow ones.

Before launch, and after it

Passing the test set is the condition for a first release, not the end of testing. We usually release to a small group first, or run the feature in the background next to the current process, and compare. Real questions from that group go into the test set, especially the ones that failed.

After launch the same set keeps running. It runs when we change the feature, and it runs when the provider releases a new model version, before we switch to it. The trace of every production request, which we describe in Log every request, is where new test cases come from.

What you have at the end

At launch you have more than a working feature. You have a test set built from your own cases, a set of graders you can trust, a number that tells you whether the feature is getting better or worse, and a history of every change and its effect. That set belongs to you and lives in your repository. If you change studios, models or providers, it still tells you whether the new setup is as good as the old one.

If you are about to build a feature and want to know what its test set should look like, write to us. It is usually the first thing we build.