How do I put an AI agent in production without it staying a demo?

Most agents never leave a notebook because nobody wrote what finished looks like. I have watched a clever thread get pasted into a shared inbox and send the same mail eleven times. The model was fine: the loop had no stop, no spend cap, and no test that the artifact existed.

Production for an AI agent is a finish line you can test, a short tool list, a runtime that can kill a runaway loop, an eval on frozen jobs, and one real desk. The model is not the product. The loop, the grants, and the score are.

What production means for an AI agent

Production means the job finishes without you in every click, and you can prove it. Chat in a notebook is research. A worker that writes a file, opens a pull request, or archives a thread, then stops, is an agent you can run because the difference is the loop you own, not the size of the model.

Anthropic’s Building effective agents, published 19 December 2024, draws the line I use. Workflows are systems where models and tools follow predefined code paths.

Agents are systems where the model directs its own process and tool use. Most teams still need the workflow. Use an agent when you cannot draw the path in advance, and when you can still write what done looks like.

Anthropic’s Building effective agents (19 December 2024) is blunt: find the simplest solution, and only add complexity when it demonstrably improves outcomes. Agentic systems trade latency and cost for flexibility.

If a script with no model would do, write the script. If the work is a conversation, use a chatbot. I do not put an agent in production to look modern.

I also do not start with a fleet. One worker with a short tool list is a job. A swarm is a shop you have not earned: extra agents add extra failure at the handoff, and production starts with one loop you can kill, score, and read.

The five steps

I ship agents in five moves. Write a done condition a script can see, allow only the tools that job needs, and run the loop in a runtime that owns retries, memory, spend, and stop. Score the artifact on a frozen set, then point the agent at one desk for a week.

1. 01

   Write the finish line

   Name the artifact. Brief.pdf in the shared drive and the vendor thread archived. If a script cannot tell that this happened, a person will have to, forever. Put a step cap and a spend cap next to the goal.

2. 02

   Give it only the tools the job needs

   List the calls: search mail, write a file, archive a thread. Each one has a schema, a timeout, and a grant. Open-ended browsing is a last resort. Extra tools are how it wanders. Search is not send.

3. 03

   Put the loop on a runtime that can stop it

   The model should not own the loop. The runtime retries a dropped connection, remembers what already ran, traces every call, fires guardrails, and kills the trace at the cap. I use one runtime for every agent so next month’s score still means something.

4. 04

   Score it before anyone depends on it

   Freeze ten to fifty jobs you already know. Run them more than once. Log every tool call. Score the artifact, not the essay. Keep a baseline. If you cannot name the score, you are still in demo.

5. 05

   Point it at one desk

   One real job, one week, a person reading the traces. Failures go into the frozen set. Then take the person out of the inner loop where the score holds. Expand only after that week is boring.

I do them in this order on purpose. A finish line without tools is a wish, tools without a runtime is a notebook, and a runtime without an eval is an invoice waiting to happen.

An eval without a desk is a lab. Skip eval and the first users become the test set. Skip the runtime and a retry storm is the first bill.

Write a finish line you can test

Name the artifact and the test that proves it exists. A PDF in a folder with the thread archived is a finish line, while write something useful is not. If a script cannot tell the job finished, a person stays in the loop, and max steps and a spend cap are finish lines too.

I start every production job by writing done in one sentence a script can check: the file is at this path, the pull request is open and the tests are green, the ticket is in this status with the note on the record, or the draft sits in this folder and the thread is archived. Vague goals produce vague loops. The model will chat until the budget dies.

Anthropic’s 2024 agents post describes the same shape. Agents plan and operate, they take ground truth from the environment at each step, and the task often terminates on completion.

It is also common, they write, to include stopping conditions such as a maximum number of iterations. I treat that cap as part of the finish line, not as ops trivia you add later.

Here is a finish line I will ship. Between Monday 00:00 and Friday 18:00, unread vendor mail becomes a one-page PDF at /out/brief.pdf, junk is archived, and the PDF has the sender, the ask, the date, and a link to the thread. A script checks that the path exists, that the file is not empty, and that those threads now carry the archived label, and if any check fails, the run is a miss.

Here is a finish line I will not ship: summarize the inbox, be helpful, draft something the team can use. Those are chat goals. They have no artifact, no test, and no stop.

People argue about the prose. The agent keeps going. That is how a notebook becomes a shared inbox incident.

Write the negative finish line too. The agent may not send mail, delete records, merge to the default branch, or move money. Those are grants the runtime does not hand over, not vibes in the prompt.

A finish line that says write the brief and never send is two rules. Keep both.

Spend is a finish line. I set a step cap and a money cap per run, and when either trips, the runtime kills the trace and records a miss. Unbounded loops are how demos become invoices, and I would rather a job stop short than retry its way through a weekend.

If only a person can tell that the work finished, that person will stay in the loop forever. That is the whole of How to run AI agents without a human, and it starts here.

Make done machine-checkable before you talk about autonomy. Autonomy without a test is a person reading chat on a Saturday.

Give it only the tools the job needs

List the calls the job actually needs, each with a name, a schema, a timeout, and a grant. Search is not send, a draft is not the list, and a branch is not production. Extra tools are how the agent shops instead of working, and open-ended browsing is a last resort, not a default.

Tool calling is the contract between the model and the world. You declare named functions.

The model picks a name and arguments. Your runtime runs that function, returns a short true result, and the agent continues.

Without tools you have a chatbot. With thirty tools you have a wanderer. I start under ten.

Each tool gets a docstring I would give a junior hire. Name, when to use it, when not to, the exact arguments, the errors it can return.

Anthropic’s 2024 post spends an appendix on this. They call it the agent-computer interface. They argue you should invest in that interface the way you invest in a human-computer interface: obvious names, formats the model has seen, no formatting overhead, tests of how the model actually uses the tool.

Their SWE-bench example is the one I quote to teams. Relative file paths broke after the agent left the root directory, they changed the tool to require absolute paths, and the model then used it cleanly.

That is production work. You do not fix that class of bug in the system prompt. You poka-yoke the tool so the wrong call is hard to make.

I split tools by permission, not by theme: read tools (search mail, fetch a file, list tickets, get a diff), write tools (create a draft, open a pull request, archive a thread, post a note), and spend tools (send, charge, deploy, merge, delete).

An agent that can read the inbox should not also be able to wire money. If one worker would need both the repo keys and the bank keys, I split the worker, or I refuse the job.

• One job per tool. search_mail is not send_mail. get_file is not write_file.
• Strict schemas. Enums beat free text. Idempotent writes, or a dry_run flag.
• Timeouts and retries as tool policy, not as prompt advice.
• Errors the model can use: not found, denied, timeout, already done. Not a stack trace.
• A short true result. Facts, ids, counts. No marketing language wrapped around the record.

Open-ended browsing is how an agent leaves the job. If the work needs a page, I give it a fetch tool aimed at a host I named, or a search that returns a capped list. A sandbox with a timeout is a tool, and a shell on the laptop is not.

Return a short true result: not found, denied, timeout, already archived, the record id. If you wrap the output in a paragraph, the model treats the paragraph as the world.

Put the loop on a runtime that can stop it

The model should not own the loop. A runtime owns retries, memory of what already ran, tracing, guardrails, spend, and stop. A notebook with an API key is still a demo, and I will not ship the loop until I can halt it.

I want one place that starts a run, steps the model, executes tools, writes memory, and ends. The brand of the model matters less than whether I can open the trace, cap the spend, and halt.

I use one runtime for every agent so a score from last month still means something. A tool that lives only in a notebook is a demo.

YouTube

Open original

LangChain walked the same runtime-and-persistence story when they shipped LangGraph 1.0 on 22 Oct 2025.

Anthropic’s 2024 guidance matches how I treat frameworks. Start with the API if you can see the prompts, and if you use a framework, understand what it hides. I will take a thin runtime I can read over a thick graph I cannot.

Tracing is not optional. The OpenAI Agents SDK tracing docs (2026) record a run as a trace of spans: model generations, function tool calls, handoffs, guardrails, and custom events.

Tracing is on by default. You can watch it in their Traces dashboard in development and in production. If you cannot open a trace and see every tool call, you cannot debug a Friday incident.

Guardrails sit in the same runtime, not in a wiki. The OpenAI Agents SDK guardrails docs (2026) split checks three ways: input on the first user input, output on the final agent output, and tool guardrails that wrap each custom function tool before and after execution. A tripwire raises and the runner halts, which is a kill switch, not a policy PDF.

Input guardrails can run in parallel with the agent or they can block. Parallel is faster. Blocking finishes before the expensive model starts, so a bad prompt does not spend tokens or fire tools, and for production writes I block.

Some writes should not fire until a person says so. LangGraph’s interrupt docs pause graph execution, persist state through a checkpointer, and wait, then you resume with a Command that carries the decision. The pause is durable, so a reviewer can come back later and the run does not replay from scratch.

I use that shape for send, deploy, merge, and spend. Search and draft run, the write that leaves the building waits, and LangGraph’s HITL middleware maps tools to approve, edit, reject, or respond.

A checkpointer is required. Without it, a pause is a crash. In production it sits on a database, not in process memory.

Retries, memory, and caps belong here too. Retry a dropped connection. Do not retry a business failure without a rule, and keep memory as what already ran plus the ids that matter, not a chat dump.

Anthropic’s 2024 post names a maximum number of iterations as a normal stopping condition. I set a step cap and a spend cap. I trip both on purpose before any desk depends on the agent.

Score it before anyone depends on it

Freeze a set of jobs you already know, run them more than once, log every tool call, and score the artifact against a baseline. A pretty trace that misses the file is a zero, and the first users are not a test set. If you cannot name the score, you are still in demo.

I freeze ten jobs that look like production as a start, and fifty is better. I write the finish line for each one and I do not change the set every week, or the score is noise. When a new failure shows up on the desk, I add it, and I do not swap the whole set to make this week look clean.

Anthropic’s Demystifying evals for AI agents, published 9 January 2026, defines an eval the way I run one. Give the system an input, then apply grading logic to what came back.

They split graders into code-based, model-based, and human. I use code first: the file exists or it does not, the tests are green or they are not, the label is on the thread or it is not.

A model judge is fine for tone and partial credit. It is a bad only-score.

A judge that grades writing will pass an agent that never shipped the work. Anthropic’s 2026 evals post is about this difficulty: the same autonomy that makes agents useful makes them harder to evaluate than a single-turn prompt. So I grade the outcome, then I read the transcript for how it got there.

Building effective agents in 2024 already said this. Add complexity only when it demonstrably improves outcomes.

New tools, new models, new prompts, all wait on the frozen set. If the score does not move, the change does not ship.

I log every tool call: name, arguments, result, latency, cost, error. Failures hide in the tools: a bad search the model then talked around, a timeout that turned into two charges.

If you only keep the final message, you will not see it. Drop secrets from the spans. Keep the names and the errors.

I keep a baseline: last week’s agent, a script, or a careful human on the same ten jobs.

If the new loop does not beat the baseline, it does not ship. Beating a human on speed while missing the artifact is not a win.

Beating last week on cost while doubling the miss rate is not a win. Name the primary score. Mine is done or not.

Run the same job more than once. Agents are not deterministic. One lucky trace is a demo, and I want the miss rate, not the best run.

When two traces diverge, I read the tool log, not the chain of thought. The chain of thought is a story. The tool log is what happened.

How to evaluate AI agents is the longer sequence: freeze the set, run competing traces, log the calls, score the artifact, keep the baseline. This step is the gate.

Production starts after the score holds, not after the demo lands. I will not point an agent at a desk I care about until that gate is boring.

Point it at one desk

One real job, one week, a person reading the traces, then take the person out of the inner loop where the score holds. Expand only after that week is boring. Do not start with a swarm or with everyone’s inbox, because autonomy is a grant you issue per job, like a permission.

I pick a job that already happens every day: vendor brief, inbox triage, a class of tickets, or a draft pull request from a spec. The finish line is already real, which means the eval set can be real.

A made-up job makes a made-up score. Production needs the ugly cases: missing attachments, duplicate threads, the vendor who writes in screenshots.

For a week, a person watches. They do not click every tool. They read the traces after the run, they catch misses, and they add those misses to the frozen set, which is How to run AI agents without a human applied to the first desk.

People stay at the edge: the goal, the grants, the review. They should not sit inside the trace unless the score failed.

Writes stay behind interrupt until that week is boring. LangGraph’s HITL pattern is the right shape: pause before send, deploy, merge, spend, then approve or edit. After the miss rate holds, I promote one write at a time.

Draft can go unsupervised before send. Open pull request can go before merge. Inbox archive can go before anything that leaves the building.

I promote a job, not the whole agent. Inbox triage can run unsupervised once it beats the baseline.

Deploy might still need a reviewer. Anthropic’s 2024 post says agents can pause for human feedback at checkpoints or when they hit blockers. I keep those checkpoints on the tools that can hurt.

When the week is boring, I expand the same job: more mailboxes of the same shape. A second desk gets the same runtime, the same score, the same grants. A fleet waits until one worker would hold permissions that should never sit together.

How to put an ops agent in production is still these five steps aimed at a queue. Noisy work needs a tighter finish line, not a looser eval.

If the first week is exciting, you are not ready. Excitement is incidents, double sends, traces you cannot read, a cap you never tripped.

Boring is the artifact landing, the miss going into the set, the person at the edge going home. Expand after boring. Not after a good demo on Thursday.

What usually breaks

Most production failures are missing pieces of the five steps, not a weak model. The finish line is missing, the tool list is too long, or the runtime cannot stop. Eval happens after launch, or one key can do everything, and I look for those before I look at prompts.

• No finish line, so it chats until the budget dies.
• Thirty tools, so it shops instead of working.
• No spend cap, so a retry storm is the first invoice.
• Eval after launch, so the first users are the test set.
• One agent with every permission, so a bad step is a bad week.
• A person in every click, so you built a slow chatbot.
• Chat eval, so a pretty miss still ships.
• A fleet on day one, so the failure is a dropped handoff you cannot read.
• HITL with no checkpointer, so an approval is a crash.
• Retries on business errors, so the customer gets the mail twice.

Split the grants too. Guardrails and interrupts do not replace a key that cannot send.

When it is ready

It is ready when it beats a baseline on a frozen set, the traces are readable, the kill switch works, and one desk has had a boring week. Pretty traces that miss the artifact are zeros, and a tight operator with a real finish line can ship in days. The slow part is reading failures, and that is the work.

I want five proofs in hand: the finish line is machine-checkable, the tool list is short and split, I have killed a run on purpose and watched the runtime halt, the frozen set has a named score against a baseline, and one person at one desk has read a week of traces without sitting inside every loop.

I also want the negative proofs: it did not send on a read grant, it did not retry a post into a duplicate, the input guardrail blocked a job it should not accept, and the interrupt paused a spend tool and resumed from a checkpointer, not from a memory. If I have not seen those, I have not tested production. I have tested the happy path.

Anthropic’s three principles from the 2024 post are a fair exit checklist: simplicity in the design, transparency in the planning steps (traces you can read), and a carefully built agent-computer interface (tools you tested).

Frameworks can get you moving. Production is when you can still see the loop.

If the score holds, promote the job, not the slogan. Take the person out of the inner loop for that job, and leave them on the writes that can hurt. Do not add a swarm because the first worker worked: the first worker worked because the finish line was real and the runtime could stop.