When agents stop

By an agent we mean software that works through a task in steps and chooses each step itself: it reads a request, looks something up, calls an API, reads the result and decides what to do next. A support agent might find an order, check the delivery status and draft a reply. A back-office agent might read an invoice, match it to a purchase order and prepare the booking.

The interesting design question is not how the agent does the work. It is where it stops. An agent with no clear edge will eventually do something expensive, embarrassing or irreversible with great confidence. This note is about drawing that edge.

Write the hand-over rules before the prompts

Before we write a single prompt, we list every action the agent could take and sort them by what a mistake would cost:

• Reading, such as looking up an order or searching the knowledge base. The agent may do this freely.
• Changes that are easy to undo, such as adding an internal note or setting a ticket’s category. The agent may do these, and every one is logged.
• Changes that are hard or impossible to undo, such as issuing a refund, sending an email to a customer, deleting data or committing money. The agent prepares these, and a person approves them.

This list is written with the people who own the process, in their words, and it becomes part of the code. The model never decides on its own which category an action belongs to.

Give the case explicit states

The state of a case must live in your system, not in the model’s memory of the conversation. We give every case a small set of named states and allowed moves between them.

received ──▶ working ──▶ done
                │  ▲
                ▼  │
        waiting_for_tool
                │
                ▼
      waiting_for_person ──▶ working   (approved or edited)
                         └─▶ closed    (rejected)
any state ──▶ failed                   (with a reason)

Explicit states make the agent’s work visible. Anyone can see how many cases are waiting for a person, how long they have waited and which ones failed. They also make the agent restartable: if a server goes down in the middle of a case, the case resumes from its last state instead of starting again or vanishing.

What sends a case to a person

We combine several kinds of triggers, because no single one catches everything.

• Rules. An amount above a limit, a customer marked as sensitive, a contract type the agent was not built for, words like “lawyer” or “complaint”. Rules are dull and reliable, and they come first.
• Failed checks. The agent’s draft fails a check: a number does not match the source, a required field is empty, a policy filter fires.
• Repeated errors. A tool keeps failing, or returns something the agent cannot interpret.
• Budget used up. The case has taken more steps or cost more than its limit allows. An agent that has not finished in eight steps is rarely going to finish well in twenty.
• The user asks. Anyone who asks for a person gets one, without an argument from the software.

We are careful with the model’s own sense of confidence. Asking the model whether it is sure produces an answer that sounds meaningful and often is not. A failed check is a much better signal than a low self-reported score.

Retries without double refunds

Tools fail. Networks time out, APIs return errors, rate limits kick in. An agent has to retry, but a careless retry is how a customer gets refunded twice.

So we separate errors that are worth retrying, such as a timeout or a rate limit, from errors that are not, such as a validation error that will fail the same way every time. Retries wait a little longer each time and stop after a fixed number. And every action that changes something carries an idempotency key: a unique id for that intended action, so that if the same refund request arrives twice, the payment system carries it out once.

Hand over with the whole context

A hand-over that says “agent could not complete the task” wastes the person’s time. They have to redo everything the agent already did. A good hand-over gives them, on one screen:

• a short summary of the request and what the agent found;
• the steps it took, with the data it used;
• the reason it stopped;
• the action it proposes, ready to approve, edit or reject in one click.

An example. The order, the customer and the limit are made up for this note.

The person’s decision is logged, and it goes back into the test set. Over time those decisions show which hand-overs were necessary and which rules were too strict.

Keep an audit trail

Every step is recorded: what the agent saw, which tool it called with which input, what came back, what it decided, and who approved what and when. This is not only for debugging. When a customer asks why their claim was handled a certain way, you can show them. When an auditor asks who authorised a payment, the answer is a person’s name, not “the AI”.

Limit what the agent can reach

The safest action is one the agent cannot take. Each agent gets its own credentials with the narrowest permissions that do the job: read access where it only needs to read, no access at all to systems it does not need. Tools are listed explicitly, and an agent cannot add new ones by itself. Limits on steps, time and spend apply to every case.

Start narrow, then widen

Our first version of an agent usually does not act on its own at all. It prepares every case and a person approves every action. That sounds slow, but it is the quickest way to learn where the agent is right. When the logs show that a type of action is approved without change, case after case, you can let the agent do that type alone, and keep a sample under review. Automation then grows from evidence instead of from hope.

If you are planning an agent and want help drawing its edges, write to us.