Log every request

On our home page there is an example answer with a trace you can open: the checks, the search, the model call, the time and the cost. Every feature we build keeps a record like that for every request. This note lists what goes into it, how we keep it safe, and what it is good for.

Ordinary application logs are not enough for an AI feature. They tell you that a request arrived and returned in two seconds with status 200. They do not tell you which version of the prompt was used, which documents the model saw, or why it decided to call a tool. Those are exactly the things you need when an answer is wrong.

The fields we record

A trace is one record per request, made of one span per step. These are the fields we treat as required.

Who and where

• a request id, which also appears in the user interface so a support person can find the trace from a screenshot;
• the feature and the environment;
• a pseudonymous user id and the customer account, never a name or an email address;
• the version of the feature, the prompt version and the configuration that was active.

What the model saw and did

• the input after any clean-up, and the query that was actually searched if it was rewritten;
• the ids and scores of the passages retrieval returned, and which of them the answer cited;
• the model, its provider and its settings;
• every tool call, with its input, its output and how long it took;
• the final output.

How it went

• each check that ran and its result: personal data, grounding, policy, format;
• tokens in and out, and the cost at that day’s price;
• time per step and in total;
• errors and retries;
• the user’s feedback, if they gave any, attached to the same record later.

{
  "request_id": "req_7f3a2c",
  "feature": "handbook_assistant",
  "feature_version": "1.8.0",
  "prompt_version": 14,
  "user": "u_51c9",               // pseudonymous
  "account": "acct_0042",
  "query": "holiday carry-over rules",
  "retrieved": [
    {"id": "handbook/4.2", "score": 0.83, "cited": true},
    {"id": "hr-update-2026-01", "score": 0.77, "cited": true},
    {"id": "handbook/4.5", "score": 0.61, "cited": false}
  ],
  "model": {"provider": "…", "name": "…", "temperature": 0},
  "checks": {"pii": "pass", "grounding": "pass", "policy": "pass"},
  "tokens": {"in": 2140, "out": 212},
  "cost_eur": 0.0019,
  "latency_ms": {"search": 240, "model": 1390, "checks": 170, "total": 1800}
}

The values above are invented for illustration. The shape is what matters.

Keep personal data out

A trace that contains everything users type is also a store of personal data, and sometimes of very sensitive data. We treat it that way from the first day.

• Remove before storing. Names, email addresses, phone numbers, account numbers and similar details are detected and replaced before the input is written to the trace.
• Store less where you can. For many features, ids and scores are enough, and the full text is kept only for a sample, or only when a user reports a problem.
• Keep it for a fixed time. Traces have a retention period, agreed with you and written down, after which they are deleted.
• Limit who can read it. Access to full traces is limited to the people who need it, and that access is logged too.
• Keep it where it is allowed to be. If your data must stay in the EU, so must the traces.
• Never log secrets. API keys, tokens and passwords are filtered out of tool inputs and outputs.

Where the traces go

You do not need a special product to start. Traces can be sent as OpenTelemetry spans into the monitoring stack you already run. There are also tools built for LLM traces, such as Langfuse, LangSmith or Arize Phoenix, some of which you can host yourself. We choose with you, based on where your data may live and what your team already knows how to operate.

What the trace is for

Collecting traces is only worth it if they are used. We use them for five things.

• Finding the cause of a bad answer. Was the right document found? Was it cited? Did a check fail and nobody looked? The trace usually narrows it to one step.
• New test cases. Every answer a user marks as wrong becomes a candidate for the test set described in Test before launch.
• Comparing versions. Because each trace carries the prompt and feature version, you can compare quality, speed and cost before and after a change on real traffic.
• Cost reports. Cost per feature, per customer and per day comes straight from the traces, as described in Limits on cost.
• Answering “why did it say that?” When a customer, a manager or an auditor asks, you can show the sources and the steps, not a guess.

Make the trace easy to open

A trace nobody can find is not much better than no trace. We put the request id somewhere the user can see it, for example in a small “report a problem” link under each answer. When a customer writes to support, the support person pastes the id and lands on the full record: the question, the sources, the checks, the answer. No developer is needed to answer “what did the assistant see?”.

For agents, the trace is longer, because one case can run for minutes and involve many steps and tools. There we add the state of the case at each step and the person who approved any action, so the trace doubles as the audit trail described in When agents stop.

When volume is high

For a feature with a lot of traffic, storing every full trace can be expensive. We then keep the metadata of every request, which is small, and the full content for a sample plus every request that failed a check, was slow, was expensive or received negative feedback. Those are the ones you will want to read anyway.

If your AI feature is live and you cannot answer “what happened?” for a given request, the trace is where we would start. Write to us.