Ask a team why their AI pilot hasn't gone live and you'll usually hear a version of "the model isn't quite good enough yet."
It's almost never true. The model is fine. What's missing is everything around it — the parts nobody demos, because a kill switch does not look impressive on a slide.
Here's the honest list of what separates a prototype from something you'd let run unattended on a Saturday.
1. An eval suite, or you are guessing
Without evals, every change to your agent is a vibe. You tweak a prompt, try three examples, decide it feels better, ship it. Two weeks later something that used to work doesn't, and nobody can say when it broke or why.
An eval suite is just a set of real inputs with known-good outcomes, run automatically against every version. That's it. The bar is lower than people think — thirty cases beats zero cases by an enormous margin — but the discipline is non-negotiable.
The critical property is that evals encode your definition of correct, not a benchmark's. "Did it extract the right entity name" is a real eval. "Does it score well on a public reasoning benchmark" tells you nothing about whether it will handle Mrs Tan's third message on Thursday.
The moment you have evals, three things change. Prompt changes become measurable. Model swaps become a Tuesday afternoon instead of a quarter. And regressions get caught by CI instead of by a customer.
2. Tracing, or you cannot debug it
When a normal service misbehaves you read the logs. When an agent misbehaves, "the logs" need to include the model's reasoning, every tool it called, what each tool returned, what was in context at each step, and how much all of it cost.
Without that, a failure report is unusable. "It gave a weird answer" — to what prompt, with what retrieved context, after which tool call returned what? An agent is a chain of decisions. If you can only see the last one, you're debugging blind.
Good tracing also changes the conversation with the client. Instead of "the AI made a mistake," you get "the retrieval step returned a stale document from March, here it is." One of those is a mystery. The other is a ticket.
3. Budgets, or it will surprise you
Agents can loop. A planner that decides it needs one more search, then one more, then one more, is behaving exactly as designed and will happily spend fifty dollars answering a question worth five cents.
So every agent gets hard ceilings: maximum steps per task, maximum tokens, maximum wall-clock, maximum spend per conversation. Hitting a ceiling isn't an error condition to hide — it's a signal to escalate to a human, with the trace attached.
Latency deserves the same treatment. An agent that takes forty seconds to answer a WhatsApp message has failed even if the answer is perfect, because the customer left. Budget for the experience, not just the invoice.
4. A kill switch, or you don't really control it
This is the one people skip, and it's the one that decides whether a client sleeps at night.
A kill switch means: a single, well-tested action that stops the agent from acting, immediately, without a deploy. Not a code change. Not "we'll comment out the cron job." A control someone non-technical can hit at 9pm when something looks wrong.
It has levels, and they matter:
- Pause a workflow — stop follow-ups going out while you investigate, leave intake running.
- Drop autonomy — the agent keeps working but every action becomes a draft awaiting approval.
- Full stop — nothing goes out, everything queues, humans take over.
Building this is not hard. Deciding to build it before you need it is the hard part, because on the day you need it you will need it in the next ninety seconds.
Why this is the actual product
There's a reason we ship evals, tracing, budgets, and a kill switch with every agent rather than offering them as a hardening phase later.
The thing you're really selling a client isn't intelligence. Intelligence is now a commodity you can rent by the token from several vendors, and it gets cheaper every quarter. What you're selling is the right to let this thing touch real customers. That right is earned entirely by the machinery around the model — the ability to measure it, watch it, cap it, and stop it.
Which is also why we keep the model layer swappable. If your entire system is a prompt tuned to one vendor's quirks, you don't have an architecture — you have a dependency. With evals in place, changing models is an experiment with a number at the end.
The uncomfortable test
Here's a question worth asking your team about any AI feature currently in progress:
If this ran unsupervised tonight and did something wrong, how long until we knew, and what would we do about it?
If the answer involves the word "hopefully," the model isn't your problem.
This is the second post in a series on shipping agents that survive production. Friday: what "autonomy level 2" actually means — a plain-English ladder for deciding how much rope to give an agent.