ServicesEvals and monitoring

Evals and monitoring

Tests that measure quality, speed and cost on every change, and tracing that shows what happens in production.

Good forAI features that are live, or about to be, and need to stay good.

Often paired withEvery other service. We build evals into all of them.

Start a project

What it is

An eval is a test for an AI feature. Monitoring is the same test, run on real traffic.

Ordinary tests check that code gives the one right answer. An AI feature answers a little differently each time, so its tests have to measure instead of compare. Is the answer correct? Is it backed by a source? Is it safe? How long did it take, and what did it cost?

We build these measures around the goals you care about, run them on every change, and keep them running in production. A new prompt, a new model or a new batch of documents then becomes a decision you make with numbers.

Typical uses

Whenever you need to know if a change made things better or worse.

  • Choosing between two models, or two prompts, on your own cases
  • Moving to a new model version without surprises
  • Checking a feature someone else built before you trust it
  • Showing a customer or an auditor how the feature behaves
  • Catching a drop in quality or a jump in cost on the day it happens
  • Turning user complaints into tests, so the same mistake does not come back

We often start here when a feature is already live and nobody can say whether it is getting better or worse.

What you get

A clear picture of how your AI feature performs, and an alarm when it changes.

The test sets

Real cases with the expected outcome and the scoring rules, kept in your repository and reviewed with your experts.

The checks

Automatic runs on every change, in your build pipeline, with a report a product manager can read.

The tracing

A record of each production request: input, context, model, output, time and cost, with personal data removed.

The alerts and reports

Thresholds agreed with you, a named owner for each alert, and a regular summary of quality, speed and cost.

What a scale does

A scale does not tell you what to buy. It tells you what you have.

Evals work the same way. They will not choose your model or write your prompt. They tell you, plainly and the same way every time, what a change did.

A report you can read

Every change produces a report like this one. It compares the new version with the current one on the same cases, and it blocks the change when a measure falls below the line we agreed.

Under it is the alert log, which tells you what happened in production this week and what was done about it.

Report240 cases

Prompt version 15 against version 14, on 240 cases

Measurev14v15Line
Correct answers88%91%85% or more
Claims backed by a source96%97%95% or more
Correct “not found” replies79%72%75% or more
Median time1.9 s1.7 s3 s or less
Cost per answer$0.0024$0.0021$0.004 or less

Result: blocked. One measure is below its line.

AlertsThis week

Alerts this week

  1. Mon 08:00Weekly report sent to the product owner.
  2. Wed 14:20Cost per answer above the line for 30 minutes. Traced to one account sending very long files. A size limit was applied.
  3. Fri 09:10Provider announced a model update. A test run is booked before the switch.

An example. The versions, the scores and the alerts are made up for this page.

What can go wrong

Evals can mislead too, if they are built carelessly. These are the traps we avoid.

Ask about an audit

Tests nobody trusts

Test cases invented by developers miss what real users ask. We build them from real requests and review them with the people who know the answers.

  • Real cases
  • Expert review
  • Versioned test sets

Chasing the score

A team tunes the feature to one number and loses the goal. We use several measures and have a person read a sample of answers every round.

  • Several measures
  • Human review
  • Goals in writing

A judge that drifts

A model that grades other models can be wrong too. We check the judge against human labels and pin its version.

  • Calibrated judges
  • Human labels
  • Pinned versions

Too many alerts

When everything alerts, nobody looks. We keep a few alerts with agreed thresholds, and each one has an owner and a written response.

  • Few alerts
  • Owners
  • Runbooks

Private data in logs

Traces can hold personal data. We remove it before storage, keep traces only as long as needed, and limit who can read them.

  • Redaction
  • Retention limits
  • Access control

How we set it up

When a feature is already live, we start by measuring it as it is. Nothing changes until we know where it stands.

For a new feature, the same steps happen before launch, as part of the build.

First week

Real traffic and complaints readwith you

First test cases writtenfrom real requests

What good looks likein writing

Second week

Tests run on the live versiona baseline

Tracing added in productionevery request

Alerts and ownersagreed

After that

Tests on every changeautomatic

Report on quality, speed and costmonthly

A baseline, the tests and the alertsabout two weeks

Questions

All questions

Which tools do you use? Open-source tracing and eval tools that you can host yourself, or the observability tools you already run. We avoid locking your test sets into a product you cannot leave.

Can one model grade another? Yes, for many measures, and it is much cheaper than a person grading every answer. We check the grading model against answers people have labelled, and keep people in the loop for the measures that matter most.

How many test cases do we need? Fewer than most teams think to start: 50 to 100 real cases tell you a lot. The set grows from there, mostly from the failures you find in production.

Will this slow our team down? A test run takes minutes. It saves the days spent arguing about whether a change was better, and the weeks spent finding out after launch that it was not.

Can you add evals to a feature we built ourselves? Yes. It is one of the most common ways we start. Your team keeps building, and we give them the tests and the numbers.

Who watches the alerts? A named owner on your side, or us, if we work on a monthly retainer. Either way, each alert comes with a short written note on what to check first.

Measure it first

If nobody can say whether your AI feature is getting better or worse, start here. Tell us what it does and who uses it.

A few lines are enough. We reply within one working day.

Start a project