How do I let an AI agent run incident and ops runbooks without a human on every step?

The pager went to a person because the runbook lived in a wiki. Someone opened the page, copied a command, waited on a graph, then copied the next command. I let an agent run that job the same way I would let a junior on-call run it: read first, write only with a grant, and page a human the moment the next step is a guess.

An ops runbook is a sequence of checks and actions with a done condition a script can see. An agent can walk that sequence if reads are free, writes sit behind a grant, and a miss pages a person. The model picks tools, the runtime owns stop, retry, and page, and people set the job and review the score.

This is the same rule I use in how to run AI agents without a human: people stay at the edge. They should not click get_logs, and they should not rubber-stamp a restart they cannot see. If you still sit inside every tool call, you built a slow runbook, not an agent.

Name the runbook as a job

Done is not a summary in Slack. It is a graph back in band, a ticket in a known state, and a replica count that matches the spec. If only a person can tell the job finished, a person will stay in every step, so write the test first and then give the agent tools that can pass it.

Most “AI on-call” demos fail here because the prompt says “fix the latency,” and latency is a symptom. The job is “p95 on checkout under 400ms for ten minutes, error rate under the SLO, ticket moved to mitigated,” and that sentence is a test. A script can poll the SLO and the ticket, so a person does not have to stare at the chart for the agent to stop.

Anthropic’s Building effective agents (December 2024) draws the line I use on the desk: a workflow is a predetermined path, and an agent is a model that directs its own tool use. A disk-full restart is a workflow with a small agentic patch (which host, which unit, did the graph recover). A novel outage with no runbook is not a job for an unsupervised loop, but you still need a finish line even when the path is not drawn.

I write the job on the ticket template before I bind a single tool: service, symptom, SLO, done condition, blast radius, and who to page if the agent stops. If I cannot fill that card, I do not have a runbook. I have a chat about production.

• A done condition a script can poll. Graph, ticket field, replica count, queue depth.
• A blast radius. One service, one cluster, one region. Not “the platform.”
• A clock. The run dies at the budget, not when the model feels finished.
• A miss rule. Unknown is a page, not another tool call.

Read-only first

The first version of the agent cannot change production. It can read metrics, logs, config, deploys, and ticket state, and it can write a diagnosis into the incident doc. It cannot restart, scale, mute, or patch, so a wrong diagnosis still leaves you with a system instead of a second outage.

Anthropic’s framework for safe and trustworthy agents (August 2025) is blunt about this split. Claude Code starts read-only in the directory it is given, and it must ask before it modifies code or systems. Users can later grant persistent permission for routine work they already trust, which is the ops pattern: read is the default, and write is a grant you earn with a score.

I bind a short read set: get_slo, get_logs, get_deploy, get_config, get_ticket, get_replica_count. Each tool has a schema, a timeout, and a service scope. Open-ended shell is not a read tool, because a shell that can cat a file can also rm it, and if the job is checkout latency the agent does not get billing logs.

Then I freeze last month’s incidents and run the agent against the same signals the on-call had. I score the diagnosis on whether it named the failing dependency, the bad deploy, or the saturation. A fluent incident summary that points at the wrong service is a miss, because the artifact is the score and the prose is not.

Only after that score holds do I talk about writes. The production path for an ops agent is the same as any other agent I put in production: a finish line, a tight tool set, a runtime that can kill it, then one real job. Read-only is that first job, and it is also the eval harness you will keep when writes exist.

Write behind approval

A write is a tool the runtime will not execute until a grant says yes. The agent proposes restart_service with a name, a reason, and evidence ids, and the runtime holds. A person, or a policy you already trust for that runbook, releases the call, because a draft is not a send and a restart is not a thought.

Anthropic’s Building effective agents (December 2024) says agents should pause for human feedback at checkpoints or when they hit blockers. On an incident, the checkpoint is every mutating call, and the blocker is any tool result that is not a known success or a known no-op. I do not let the model vote on whether the pause was needed, because the tool definition carries the pause.

The same August 2025 Anthropic note is useful here: humans should keep control before high-stakes decisions. Their expense example is a subscription the agent wants to cancel, and mine is a pod the agent wants to kill. Both are writes that wait, and persistent permission comes later for a runbook that already holds on eval, inside a blast radius you can name.

I split writes the way I split on-call roles: restart_unit is not restart_cluster, scale_replicas is not scale_all, and mute_alert is not mute_service. Each one has an idempotency key, a dry_run flag, and a maximum count per incident. An agent that can restart a unit should not also be able to drop a table, which is why a fleet exists: permissions that should never sit in one body.

Approval is not a Slack thumbs-up on a paragraph. It is a signed yes on a tool call with arguments the runtime will actually run. If the human cannot see the exact command, the exact host, and the evidence that justified it, the approval is theater, so I show the planned call and the last read that supports it, then I wait.

Page on miss

A miss is any moment the agent cannot prove the next step: unknown result, timeout, denied write, budget hit, or a done condition still red after allowed writes. The runtime stops the loop and pages on-call. The model does not get a turn to try one more thing, because that extra thing is how a diagnosis becomes a blast.

Paging is not a tool the model debates. It is a stop rule. Anthropic’s December 2024 agent pattern includes stopping conditions such as a maximum number of iterations, and I add the ops versions: a spend cap, a step cap, a wall clock, and a list of results that mean stop, with the page carrying a packet instead of a chat dump.

NIST’s AI Risk Management Framework 1.0 (January 2023) is the shape I keep on the wall even when the work is a single runbook. Map the context and the blast radius, measure the agent on frozen incidents, manage the grants and the page, and govern who is allowed to promote a runbook from read-only to write. The framework is voluntary, and the outage is not.

I would rather page a person at minute four than let the agent wander until minute forty. On-call already knows how to take an incident, and they do not know how to unwind a loop that restarted the wrong pool because a log line looked like a crash. The miss is the feature, so treat it as a successful handoff, not as the agent failing a vibe test.

Turn the wiki into tools

A pasted runbook in the prompt is not a contract. The model will skip a check, invent a flag, or run step four before step two. Encode each check and each action as a named tool with a JSON schema, treat the tool description as the prompt, and be exact: search is not send, and read is not restart.

Anthropic’s December 2024 appendix on tools is the part operators skip. They spent more time on the agent-computer interface than on the overall prompt when they built their coding agent, and relative paths failed once the working directory moved, so they required absolute paths. Ops has the same class of bug: “restart the api” is not an argument, while service, cluster, unit, reason, and evidence_id is.

I keep the tool list short. A checkout-latency runbook might get ten tools: get_slo, get_logs, get_deploy, get_error_budget, get_dependency_status, get_replica_count, restart_unit, scale_replicas, update_ticket, page_oncall. If I need billing, fraud, and kube-system in the same worker, I do not have a runbook but a platform intern with root.

YouTube

Open original

LangChain walked the same graph-as-runbook idea on 17 Jan 2024.

Return values must be facts the next thought can use: not_found, denied, timeout, already_at_desired_state, or the record. Do not wrap a 500 in a paragraph, and do not let the tool apologize. Ground truth at each step is the other half of Anthropic’s 2024 agent loop, because the agent assesses progress from tool results, not from its own narration.

Idempotency lives in the tool, not in the prompt. restart_unit with the same incident id and unit id must be a no-op the second time. Temporal’s 2026 activity documentation is explicit that activities that perform writes should be idempotent, because the activity may run more than once even when the workflow observes a single completion, and a runbook that restarts or pages without a key will double-fire the first time a worker dies after success.

Put retries in the runtime

The model should not retry a restart because the last call looked incomplete. The runtime retries a dropped connection, and it does not retry a business failure. Mix those two and you get a retry storm that looks like healing and is actually load.

Temporal’s 2026 retry policy documentation is the mechanism I copy even when the workflow engine is not Temporal. Activities retry by default, and workflows do not. Default activity retry is exponential backoff with a 1-second initial interval, a 2.0 backoff coefficient, a maximum interval of 100 seconds, and unlimited attempts unless you cap them, which is correct for get_logs and wrong for restart_unit.

I set writes to a small maximum attempts, and I mark permanent failures as non-retryable. Permission denied, unknown cluster, already_at_desired_state, and invalid input are Application Failures with nonRetryable set. Temporal’s docs say permanent failures will not fix themselves on retry, so they need different input or different logic: surface them, page, and do not let backoff hide a bad grant.

Reads can use the default backoff. A metrics API that 503s for three seconds should not wake a person, and a restart that 503s because the control plane is itself the incident should not keep hammering. I use Temporal’s fast-then-slow idea in spirit: a short burst for transient errors, then a long interval or a stop, and the stop is the page.

Do not retry inside the tool, because Temporal warns that inner retries lengthen the timeout you actually need, hide failure metrics, and make the UI harder to read. Put one timeout and one policy on the activity so the agent sees the final result. If that result is a miss, the runtime pages, and the loop stays boring, which is the point.

Trace the run like a deploy

If you cannot replay the tool calls, you cannot trust the agent on the next page. I log every call: name, arguments, result, latency, cost, error. Pretty traces that miss the finish line are zeros, so the incident doc is an output and the trace is the record.

OpenTelemetry’s 2026 GenAI semantic conventions are still marked Development. I use them anyway so a runbook sits on the same map as the service it touches. The conventions define events, exceptions, metrics, model spans, and agent spans, with operation names that include invoke_agent, execute_tool, invoke_workflow, and plan, and I want those names in the exporter, not a private string that only the chat UI understands.

On the agent span I set gen_ai.agent.name to the runbook and gen_ai.operation.name to invoke_agent, and I keep a conversation id that matches the incident id. Child spans are execute_tool with the tool name. Token usage goes on gen_ai.usage.input_tokens and gen_ai.usage.output_tokens so cost is a first-class signal, and if the tools arrive through MCP I still want execute_tool on the span, not a silent sidecar.

The trace has to answer the questions on-call will ask at the handoff: what did it read, what did it want to write, what was denied, and why did it page. If the answer is buried in a 40-turn chat, the person will ignore the agent next time. I keep the span attributes boring and the packet short, because correlation with the existing service traces is the whole reason to use the convention.

Measure is the NIST function most teams skip. Freeze the incidents, run the agent, and compare diagnosis and time-to-mitigate against last week’s agent and against a careful human. If the new loop does not beat the baseline on the frozen set, it does not get write grants, because the first live page is not a test set.

How I auto-run a runbook

The path is linear on purpose. Skip a step and you are back to a person clicking through a wiki, only now the clicks are hidden inside a model. I use the same six moves on every runbook I promote.

1. 01

   Write the runbook as a job

   Name the service, the symptom, the SLO, and the finish line a script can poll. Put the blast radius and the miss rule on the same card. If you cannot fill the card, stop, because you are not ready for an agent.

2. 02

   Bind read tools only

   Metrics, logs, config, ticket, deploy history, replica count. Scope each tool to the service on the card, with no shell and no open internet. The agent may write a diagnosis and may not change the system.

3. 03

   Score it on frozen incidents

   Replay a month of pages with the same signals on-call had. Grade the diagnosis against the postmortem, not against fluency. Log every tool call, and keep last week’s agent as the baseline.

4. 04

   Add writes behind approval

   Each mutating tool is its own grant, with an idempotency key and a dry_run flag. The agent proposes the exact call, and the runtime waits. A person or a policy you already trust for that runbook releases it.

5. 05

   Page on miss

   Unknown result, timeout, denied write, budget hit, or a done condition still red: stop and wake on-call. The page is a packet. The model does not get another turn.

6. 06

   Promote one runbook at a time

   A disk-full restart can go unsupervised after the score holds. A schema migrate, a failover, or a customer wipe cannot. Autonomy is a grant, like a permission, earned per job and not flipped on for the org.

That is also how to put an ops agent in production. The runbook is the first job, not a platform. One desk, one week, a person reading traces, and you expand only after the week is boring, because a swarm of specialists on day one is how the handoff drops and nobody can say which worker held the restart key.

Split the grants

One agent with read, write, and spend is a worker with root. I split the body before I split the personality. The reader can see production, the writer can change a unit it was shown evidence for, and the pager is the runtime, so those three should not share a credential.

This is the case for a fleet: I do not add specialists because a slide said multi-agent. I add them when one worker would hold permissions that should never sit together (research and deploy, diagnosis and restart, ticket updates and cloud spend), which is the split in multi-agent orchestration. Extra agents add extra failure at the handoff, so the default is still one operator with a short tool list.

Persistent permission is the last grant, not the first. Anthropic’s August 2025 note says users can grant persistent permission for routine tasks they trust, and I copy that. After a restart_unit runbook holds on eval, I let the runtime execute that one tool without a person, inside a cap, for that service, and I do not promote mute_all because restart_unit went well, because grants do not travel.

Spend is its own grant. Token budget, activity attempts, and how many units this incident may restart are numbers on the job card, and the model does not raise them. Unsupervised agents fail as retry storms, so the cap lives in the runtime, the same as when I run AI agents without a human in the inner loop.

Hand off a miss, do not narrate it

When the agent pages, the person who wakes should not have to mine a chat. They should get a packet: the finish line, the evidence ids, the next action the agent would have taken, and the limits left. That is the same handoff I use between agents, because a chat log is how the second reader starts the job over.

The outgoing run is not done until it writes the packet: goal unchanged, evidence as file ids, span ids, and log query ids rather than a vibe, next action as one step, and limits as spend, grants, and deadline. If evidence is missing, the agent calls a read tool before it pages. It does not invent a recap to look complete.

That packet is the same one I use in How to hand off an agent to a human.

Shared memory is a pile of logs, and a handoff is a decision. On-call can open the traces, but they should not have to, because the packet tells them whether to retry the proposed write, to change the job, or to take the incident fully. An evening diagnosis can be a morning write in another region without anyone re-asking what broke.

I treat a clean page as a pass on eval. The agent found the edge of its grant and stopped. A dirty page is a fail: it wrote anyway, it omitted the evidence, or it left the ticket in the wrong state, so score the handoff the way you score the mitigate, because both are artifacts.

What I keep a human on

Some runbooks should never auto-write: failover across regions, a schema migrate, a customer data delete, and anything you cannot undo in the same window you can page. The agent can still read, draft the change, and wait. The person is not in every step, only on the step that can take the company down.

I also keep a human on runbooks with no test. If the done condition is “the customer feels better,” you do not have a job an agent can finish. Use a script if the path is fixed, use a person if the path is political, and use an agent when you cannot fully draw the path and you can still write what done looks like.

Transparency is the other reason to keep a person at the edge. Anthropic’s August 2025 framework says humans need to see the plan, or they cannot tell whether the agent is on the job. Too little detail and you cannot intervene, too much and you ignore it, so I show a live checklist of tool calls and the current guess at the finish line, and on-call can stop the run without having to drive it.

The destination is a runbook that runs: read-only first, write behind approval, page on miss. If that loop is boring for a week, you earned the next grant. If it is exciting, keep the person on the write.