Limits on cost

Ordinary software costs roughly the same whether a user clicks once or a hundred times. An AI feature does not. Every request is billed by the amount of text that goes into the model and comes out of it, and that amount depends on things that change every day: how long the conversation is, how many documents retrieval adds, how many steps an agent takes, how often a call fails and is retried.

None of this is a reason to avoid AI features. It is a reason to treat cost as a property of the feature that you design, test and watch, like speed or accuracy. Here is how we do it.

Know the unit you are paying for

“The OpenAI bill” is not a useful number. The useful numbers are the cost of one answer, the cost of one active user per month, and the cost of each feature per month. You can only know them if every request records how many tokens went in and out, which model handled it, and what that cost at the price of the day.

We log this on every request, next to the user, the organisation and the feature. Once it is there, questions like “what does the summary button cost us per customer?” take a minute to answer instead of a week.

Estimate before you build

You can estimate the running cost of a feature before writing it, and you should, because the estimate often changes the design. The arithmetic is simple: the number of active users, times the requests each makes in a month, times the tokens per request, times the price per token. The hard part is being honest about the middle two numbers.

A rough example, with every number made up: 2,000 active users, 30 questions each per month, 3,000 tokens in and 300 out per question. That is 180 million tokens in and 18 million out per month. Multiply by your provider’s prices and you have a figure to put next to the value of the feature. If the figure is uncomfortable, it is much cheaper to change the design now (fewer passages, a smaller model for simple questions, a shorter conversation memory) than after launch.

We then check the estimate against the first weeks of real use, because real users always behave differently from the spreadsheet.

Limits at three levels

A single limit is not enough, because costs go wrong in different places. We set them at three levels.

Per request

A maximum size for what goes in and what comes out. For agents, a maximum number of steps and tool calls. This level stops the expensive accidents: a pasted 200-page document, a conversation that has grown for a week, an agent that keeps calling the same tool because it misreads the result.

Per user

A daily or monthly budget for each user or each customer account, with a soft limit that warns and a hard limit that stops. This level protects you from one heavy user, a script someone wrote against your API, or an account that was taken over.

Per feature

A monthly budget for each feature, with alerts at half, at eighty percent and at the limit. This level is for planning. It tells you early in the month that usage is growing faster than the plan, while there is still time to decide what to do.

# limits.yaml — an example; the numbers are placeholders
request:
  max_input_tokens: 12000
  max_output_tokens: 800
  agent_max_steps: 8
user:
  daily_budget_eur: 0.50
  soft_limit_at: 80%
feature:
  handbook_assistant:
    monthly_budget_eur: 400
    alert_at: [50%, 80%, 100%]
prices:            # kept in config, never in code
  source: providers/prices.yaml
  checked: weekly

Decide what happens at the limit

A limit without a plan for what happens next turns into an outage. For each limit we decide the behaviour in advance, with you. A request that is too long can be shortened, or the user can be asked to narrow it. A user over the daily budget can get a smaller, cheaper model for the rest of the day, or a clear message saying when the feature is available again. A feature over its monthly budget might keep working for paying customers and pause for trial accounts.

What must never happen is a silent failure: a spinner that never ends, or an empty answer with no explanation.

Spend less without answering worse

Many AI features can cost less without answering any worse. These are the places to look first:

• Shorter prompts. Prompts grow over time as people add rules for each bug. A yearly clean-up often removes instructions that no test depends on.
• Provider prompt caching. Most providers charge less for the part of a prompt that repeats exactly between requests. Putting the stable part first, and the changing part last, makes that discount apply.
• Fewer, better passages. A document search that sends twenty passages to the model, where five good ones would do, pays for fifteen it does not need. Better re-ranking is usually cheaper than a bigger context.
• Routing by difficulty. Not every request needs the largest model. A small model can classify the request first, and only hard ones go to the expensive model.
• Caching answers, carefully. Identical questions can reuse an answer, but only where permissions and freshness allow it. A cached answer must never cross from one customer to another.

Every one of these changes can also make answers worse. That is why we never make them without the test set from our note Test before launch. The test run reports quality and cost side by side, so a saving that costs accuracy is visible before it ships.

Watch it every day

Cost belongs on the same dashboard as errors and response times: cost per feature per day, cost per active user, and the most expensive requests of the day with a link to their trace. An alert fires on a sudden jump, which is almost always a bug, a loop or misuse rather than real growth.

Prices change, and so do models

Providers change prices, release cheaper models and retire old ones, often with short notice. Keep the price table in configuration rather than in code, check it on a schedule, and run the test set against a new model before you switch to it. Then a price drop becomes a saving you can take safely, and a model retirement becomes a planned change instead of an emergency.

If you have an AI feature in production and do not know what one answer costs, that is the first number we would find for you. Write to us.