How do I run AI agents without a human watching every step?

“Human in the loop” is often a way to skip eval. If a person must approve every tool call, you built a slow chatbot. The point of an agent is that the person is not in every loop.

Without a human here means without a person in every click, not without a person at the edge.

I run agents while I am off the desk. That only holds because the runtime owns the budget, not me. Tools, retries, and stop rules are the job, and if any of those three is missing, I stay in the chair.

Unattended work is a budget you write before the first run: the tools the agent may call, how many times it may retry, and the rule that stops the loop. People set that budget and read the score later. They do not click every tool.

Unattended is a budget of tools, retries, and stop rules

A model that can keep calling tools will not stop itself. The runtime must know which tools exist, what a retry is for, and when the job is over. Those three numbers are the job, not personality in the prompt.

Anthropic’s Building effective agents, published 19 December 2024, draws a hard line between workflows and agents. A workflow follows a path you drew in code. An agent is a model that directs its own tool use.

Agents fit open-ended problems where you cannot predict the number of steps. The model may run for many turns. You have to trust its decisions inside a box you sized.

That paragraph is the whole argument. Autonomy is not “leave it alone.” Autonomy is “I trust this loop inside limits I can name.”

Anthropic also writes that agents should pause for human feedback at checkpoints or blockers, and that stopping conditions such as a maximum number of iterations are common. The stop is not optional decoration. It is how you stay in control when the path is not hardcoded.

I treat the budget as three lists you can print on one page.

• Tools: named functions with schemas, timeouts, and permissions. Search mail is not send mail. Read a runbook is not restart a service.
• Retries: a dropped connection may retry. A “already refunded” error may not. Retrying a business failure is how you double-send.
• Stop: done, budget exhausted, a forbidden call, or a tripwire. The model does not get a vote on the spend cap.

If you cannot write those three, you are not ready to leave the room. Write them. Then take the person out of the inner loop.

I put this layer on one runtime so a score from last month still means something. A tool that lives only in a notebook is a demo. An unattended agent that cannot be killed is an incident waiting for a quiet hour.

People stay at the edge of the loop

A person still writes the job, signs the grants, and reads the weekly score. That is the edge of the loop. Sitting inside the trace clicking Approve on every search is a queue with extra latency, not oversight you can score.

The inner loop is think, call a tool, read the result, repeat. The outer loop is the job, the grants, and a look at what shipped. Unattended means the person left the inner loop, not that the job has no owner.

I set three things before a run ever starts: a finish line a script can test, separate grants for read, write, spend, and deploy, and a score I read after a batch of jobs rather than after every token.

NIST’s AI Risk Management Framework 1.0, published January 2023, organizes that work as four functions: Govern, Map, Measure, and Manage. People govern. They map which system is allowed to do which job.

The runtime measures and manages the run. Mixing those layers is how a “human in the loop” policy becomes a person rubber-stamping tool calls they cannot actually judge in time.

A reviewer who must approve every search will approve them. They have a queue and a clock, so they will click through. That is not a control.

A reviewer who sees a weekly fail rate, a spend spike, and a handful of traces that missed the artifact is doing the job. Put the person where judgment lives. Take them out of the path where the next click is a habit.

On my own desk the edge looks ordinary. I write “vendor threads labeled, junk archived, brief.pdf in the shared folder.” I grant search and label and write-file. I do not grant send.

I look at the score on Friday. If the score holds, I do not sit in the trace. If the score drops, I read the tool log.

Done has to be something a script can see

If only a person can tell that the work finished, the person will stay in the loop forever. Write a check a script can run: a file, a label, a merged pull request, a closed ticket, a webhook. Then the runtime can stop without asking anyone.

“Handle the vendors” is not a finish line. “PDF in /out/brief.pdf, junk archived, the rest labeled” is. Vague goals force a person to interpret, and precise goals let the loop end.

“Keep the site up” is not a finish line either. “Pager closed, runbook step recorded, the service answers on the health check” is.

Anthropic’s same 2024 note says agents need ground truth from the environment at each step, such as a tool result or a test, to assess progress. That is the check. The model’s last message is not.

Pretty prose about a missing file is still a miss. I score the artifact, not the essay.

A machine-checkable done condition does two jobs. It tells the runtime when to stop on success. It tells eval when a run is a pass.

Without it, you cannot promote a job to unattended, because you have no way to know the job held while you were gone. You will keep a person in the loop as a sensor. That person is slow, and they will drift.

• A test, a hash, a row in a table, a label, a closed ticket. Pick one you can query.
• Idempotent writes so a second run does not create a second artifact.
• A timeout that is a fail, not a maybe. “Still thinking” is not done.
• A dry run flag on anything that spends or sends, so you can score without side effects.

I refuse to take a person out of a job whose done condition is “looks good.” If you cannot write the check, you do not have an unattended job yet. Keep the person in until the check exists, because that is cheaper than a public retry storm.

Give the agent a short, split tool list

Unattended does not mean every API. It means the few tools the job needs, split so a bad step cannot become a bad week: search is not send, a draft is not the list, and a branch is not production.

Anthropic’s 2024 post spends as much time on the agent-computer interface as on the prompt. Tool names, parameters, and docs are the prompt the model actually follows. Give the model a blunt, narrow surface.

Use absolute paths instead of relative ones, prefer enums over free text where you can, and keep one job per tool. I start under ten. Each extra tool is a way to wander, and a way to spend tokens on shopping instead of the artifact.

The split that matters for unattended work is permission, not poetry. An agent that can search the inbox should not also be able to wire money. An agent that can open a pull request should not merge to production on the same grant.

If you widen the tool list to “make it more autonomous,” you did not add autonomy. You added ways to fail that you cannot see until the invoice arrives. The grants stay.

I also refuse catch-all tools. “Run any shell command” is a hole, and “open the internet and figure it out” is a last resort. Name the call, give it a schema and a timeout, and if the model cannot do the job with that list, the job is too big.

Cap retries, turns, and spend

Runaway retries are how unsupervised agents fail in public. The runtime kills the trace at the cap: a dropped connection may retry, a business failure may not. Turns and spend are numbers you wrote before the run, not a feeling during it.

The OpenAI Agents SDK (2026) is explicit about the loop: the runner calls the model, executes tool calls, and repeats, and if the run exceeds max_turns it raises MaxTurnsExceeded. You can pass max_turns=None and disable the limit. I do not, because an unattended job without a turn cap will keep calling tools until spend, not until the job is done.

Retries need a split as sharp as the tool list. Network errors, timeouts, and 429s may retry with backoff. “Already refunded,” “already archived,” and “already closed” may not, because retrying those creates duplicates.

Anthropic’s compounding-error point is the reason the cap exists. Each extra model turn is another chance to pick the wrong tool, misread a result, or spend. Latency and cost rise with autonomy.

That trade is worth it when the path cannot be drawn in advance. It is not worth it as an unbounded loop on a cron job. I set a spend ceiling per run and per day, the runtime stops, and I get a trace instead of a surprise.

• max_turns for the agent loop. A number. Not “until it feels done.”
• A money cap per run and per day, enforced outside the model.
• Timeouts on every tool. A hung call is a fail, not a wait.
• Retry only the errors you listed. Everything else stops and logs.
• A dead-letter for traces that hit the cap, so a person can read them later.

Guardrails trip without a person

A guardrail is a check the runtime runs so a person does not have to. Input, output, and tools each get a check. If it fails, a tripwire fires and the run stops, which is how you leave the room without leaving the door open.

The OpenAI Agents SDK documents three families. Input guardrails run on the first user input, and only for the first agent in a chain. Output guardrails run on the final output, and only for the last agent.

Tool guardrails wrap function tools and run on every invocation of that tool: input checks before execution, output checks after. If you only put checks at the chat boundary, a specialist in the middle of a handoff can still call a tool you never wanted.

A tripwire is the stop. If tripwire_triggered is true, the runner raises InputGuardrailTripwireTriggered or OutputGuardrailTripwireTriggered and halts. That is an exception the loop cannot talk its way around, which is the difference between a policy and a paragraph.

The OpenAI Agents SDK Guardrails guide (2026) is blunt: if the guardrail detects malicious usage, it can immediately raise an error, saving time and money.

Input guardrails have two execution modes. Parallel is the default: the guardrail runs beside the model, which is faster, but the model may already have spent tokens or called tools before the tripwire fires. Blocking mode finishes the guardrail first.

If it trips, the agent never starts. For unattended work that can send or spend, I block, because latency is cheaper than a side effect you cannot undo. For a read-only research job, parallel is fine.

Tool guardrails are the ones I care about once the person is gone. They can skip a call, replace the output with a message, or raise a tripwire.

I use them to block secrets in arguments, to refuse a send to an address that is not on the allowlist, and to redact a token that showed up in a tool result. The check sits on the tool, so every caller hits it. You do not hope the manager agent “remembers” the policy.

Output guardrails always run after the agent completes, so they cannot stop a tool that already ran. Put expensive or dangerous work behind a tool guardrail or an interrupt, not behind a last-second read of the essay.

I write guardrails as boring functions: allowlists, schemas, “this label is one of four enums,” “this recipient is on the domain,” and “this payload does not contain a key that starts with sk-.”

A second cheap model can screen for off-policy requests, which is the SDK’s own example. I still want a deterministic check in front of send and spend. Models as guards are useful, but they are not the lock on the door.

Interrupt only the irreversible step

An interrupt is a pause you planned, not a person watching every tool call. Search can run unattended while send, spend, and deploy wait. The graph saves state, waits, and resumes when a decision arrives, and if you interrupt every tool you are back in the inner loop.

LangGraph’s interrupt() call is the mechanism. You place it in a node, pass a JSON-serializable payload, and the runtime saves graph state through a checkpointer. The run waits indefinitely until you resume with Command(resume=...).

The resume value becomes the return of interrupt() inside the node. You need a durable checkpointer in production and a thread id so the runtime knows which state to load. That is how a pause survives a process restart.

YouTube

Open original

LangChain showed LangGraph: Persistence on 5 Feb 2024: the checkpointer that makes an interrupt survive a restart.

LangGraph’s Interrupts docs (2026) describe the pause I use: when an interrupt is triggered, LangGraph saves graph state through its persistence layer and waits until you resume.

The docs are clear about what this is for: approval before a critical action, review and edit of a draft, a pause inside a tool such as send_email, and validating input. The pattern I use is the first one. Interrupt before the side effect, not after.

Search, retrieve, draft, open a branch. Then pause if the next call spends money, emails a list, or ships to production. The person sees the payload, not the whole trace.

Approve, edit, or reject, then resume. The agent continues. It does not start the job over.

LangGraph also warns that static interrupts (interrupt_before and interrupt_after on compile) are for debugging, not for human-in-the-loop. Use interrupt() when the pause is part of the job.

Because resume re-runs the node from the start, side effects before the interrupt must be idempotent. Do not insert a row, then pause, then insert the row again on resume. Upsert, or put the write after the decision.

The OpenAI Agents SDK has a parallel idea: human review pauses a run so a person or a policy can approve a sensitive action, while guardrails validate automatically. Use guardrails to block. Use review to decide.

I do not make a person approve a search. I do make a person, or a narrow policy, approve a send to more than one outsider, a payment over a threshold, or a production deploy. The interrupt is the grant you did not want to give the model.

If the evening run finishes research and the morning agent must continue, that is not an interrupt. That is a handoff. The outgoing agent writes a packet: the original finish line, evidence as file ids, the next action, limits left.

The incoming agent continues from the packet. I wrote how to hand off an agent to a human for that packet. Do not dump a chat log and hope.

Promote one job at a time

Autonomy is a grant, like a permission. Inbox triage can go unsupervised before deploy does. You promote a job when the finish line is checkable, the tools are split, the caps hold, and the score beats a baseline on frozen work, and you do not promote the company.

I start with jobs whose blast radius is a few minutes of a person’s time. A wrong label on a vendor thread can be moved back.

That is how to auto-triage an inbox: search, label, archive junk, write a brief, stop. Send stays off the grant until the labels are boring. The person reads a sample of traces, not every thread.

Ops runbooks are next when the steps are already written and the health check is real. How to auto-run ops runbooks is the same budget in a different room: read the runbook, call the named steps, stop on the check.

Restart and delete stay behind an interrupt until the runbook has been dull for a while. A runbook with a finish line is a better agent job than a blank “keep the site happy.”

1. 01

   Pick one desk job with a reversible miss

   Mail labels, a research file, a draft pull request. If the agent is wrong, a person can undo it in one step. Do not start with the wire or production.

2. 02

   Write the budget and the check

   Tools, retries, stop. A script that can see done. Guardrails on input, output, and the write tools. An interrupt on anything you cannot revert.

3. 03

   Score it while a person still watches the traces

   Freeze last week’s jobs. Run them more than once. Log every tool call. Compare against a person or a script. Pretty misses still count as zeros.

4. 04

   Take the person out of the inner loop

   They keep the grants and the Friday score. They do not click search. If the score holds for a week, widen the hours, not the tools.

5. 05

   Only then promote the next job

   A second job gets its own budget. Do not copy send from triage onto deploy. Autonomy does not transfer.

Split only when one worker would hold keys that should never sit together, such as research and deploy. Extra agents add extra failure at the handoff. Promote jobs, not a swarm.

Watch traces after the run, not chats inside it

Once the person is out of the inner loop, the remaining human work is reading tool logs, spend, fail rate, and the artifact. Intervene when the score drops. Do not sit in the chat and grade tone, because tone is not the job.

I freeze ten to fifty jobs that look like production. I run them more than once. I log every tool call: name, arguments, result, latency, cost, error.

The winner is the trace that produced the artifact, not the one that explained itself. This is the same method as how to evaluate AI agents. Unattended work without that freeze is a demo you left running.

Anthropic’s 2024 post repeats the obvious because teams skip it: measure, then add complexity only when it improves outcomes. Guardrails, interrupts, and extra tools all cost turns. If a script or a workflow with a gate hits the finish line, keep that, because an unattended agent is for the jobs whose path you cannot draw and whose miss you can still bound.

When the score drops I do not add a human to every step. I read the log. Usually the tool list grew, the retry rule started retrying a business error, or the done check was never real.

I tighten the budget and re-run the freeze. Only if the job itself changed do I put a person back in the inner loop, and I treat that as a demotion. The job can be promoted again when the score holds.

I find out whether the finish line was real by running it on my own day: mail, research, code, eval, ship. If Thursday does not match Monday, the grants were too wide or the check was never real.

A person still owns grants and new jobs

Unattended is not unowned. A person still writes new jobs, signs new grants, and pulls the kill switch. The model does not promote itself, the runtime does not widen its own tool list, and that line is the safety story once the inner loop is empty.

Keep a person on anything whose miss you cannot undo in one step: money past a threshold, a customer list, production deploys, deletes. Keep them on the first week of any new job, even a small one, and on the weekly score.

A passing eval on triage does not license deploy. I will let an agent draft mail after the score holds, and I will let it open a pull request. I will not let it send the list or merge to production on the same grant until that job has its own freeze, its own caps, and an interrupt I am willing to sit on.