Notes from the lab

What we have learned building AI features, written down for product teams who are about to build their own.

Written by the people who buildEach note comes from work we do every week: tests, search, cost, agents and logs.

Plain and practicalNo news round-ups and no predictions. If a note does not help you build something, we do not publish it.

All notes

Newest first. Each note stands on its own, so start with the problem you have today. Every note ends with what we would do first, and you are welcome to ask us about it.

22 October 2026Log every requestThe fields we record for every request to an AI feature, how we keep personal data out of the logs, and what the trace is for once it exists.
22 October 2026Test before launchHow we test an LLM feature before it reaches users: test sets built from real cases, three kinds of graders, and a regression run on every change.
22 October 2026The first callThe questions we ask in the first call about an AI feature: the goal, the users, the data, the rules, and whether AI is the right tool at all.
22 October 2026Limits on costAn AI feature costs little per request and an unknown amount per month. How to set limits per request, per user and per feature, and cut cost without cutting quality.
22 October 2026When agents stopAn agent needs a clear edge where it stops and a person takes over. How we design states, hand-over rules, retries, limits and the audit trail.
22 October 2026Why search failsMost wrong answers from document search are not the model’s fault. Five ways retrieval fails, how to tell them apart, and what fixes each one.

Why we write these

We write these notes for teams who are about to build their own AI feature.

Most of what goes wrong with these features is ordinary and avoidable: a test that was never written, a document that was never updated, a limit nobody set. If a note describes a problem you have now, write to us and we will tell you what we would do first.

Start a project