How do I put an AI ops agent into production on real systems?

PagerDuty is not a chat window. If I wrap kubectl in a tool because the demo looked calm, I have not put an ops agent in production. I have given a language model a way to change a live system.

An ops agent in production is a worker with a named identity, a short allowlist, a dry-run on every write, a score on frozen incidents, and write tools only after those four hold. The grants are the product, not the model.

I wrote How to put an AI agent in production for the general case: a finish line, a short tool list, a runtime that can kill the loop, an eval, split permissions, then one desk. This guide is narrower. The desk here can page you, and the artifact is a changed system, not a PDF.

What is an ops agent covers the job. How to auto-run ops runbooks covers the path when the path is already drawn. Here I put the worker on real systems without handing it the cluster on day one, and identity, dry-run, and eval come before write tools.

What production means on real systems

Production for an ops agent is a set of grants a script can check. The agent has its own identity, the surface it can touch is named, writes plan before they mutate, and a frozen set of incidents has a score. A clever trace in a notebook is still a demo.

Ops work already has a finish line if you write it down: replica count is three, the canary is at ten percent, the alert is silenced with a ticket id, the node is drained and the replacement is Ready. If a script cannot see that this happened, a person will stay in the loop forever.

NIST’s AI Risk Management Framework 1.0 (January 2023) is voluntary. The core is still the right order for this job: Govern, Map, Measure, Manage.

Govern is who the agent is and who owns the grants. Map is the surface: clusters, accounts, runbooks, tickets. Measure is the eval, and Manage is what you do when the score drops.

The failure I see most is a shared human identity. Someone pastes a cloud key into a .env, points a chat at the cluster, and calls it an on-call bot. When it pages at the wrong replica, the audit log names a person who was not on the call.

Production starts when the actor in the log is the agent, and the grant in the log is narrower than the person who hired it.

A demo acts as a human session, calls whatever the SDK loaded, writes live, and proves itself with a pretty trace. Production names a workload identity, allowlists hosts, dry-runs until the score holds, and is done when one class of write is boring for a week.

The five steps

I put an ops agent in production in a fixed order: identity, allowlist, dry-run, eval, then write tools. I do not open a mutating API because the prompt is better this week. Skip a step and the first live page is the test set.

1. 01

   Give the agent an identity

   Issue a workload identity the agent authenticates as. Do not share a human’s keys, a laptop SSO cookie, or a cluster-admin role.

2. 02

   Allowlist the surface

   Name the hosts, APIs, namespaces, and commands. Close everything that is not on the list. Read is not write.

3. 03

   Dry-run every write

   Every mutating tool returns a plan first. Nothing in the live system changes. If you cannot dry-run it, it is not a tool yet.

4. 04

   Eval on frozen incidents

   Replay last quarter’s pages in a sandbox. Grade the outcome, log every tool call, and keep a baseline.

5. 05

   Then write tools

   Open one class of mutation after the score holds. Watch it for a week, then decide on the next grant.

Teams usually write the kubectl wrapper first because it looks like work, then spend a month adding “please be careful” to the system prompt. Prompts are not grants. Anthropic’s Building effective agents (19 December 2024) is blunt about complexity: add it only when it demonstrably improves outcomes.

Give the agent an identity

An ops agent that acts as a person is already a breach waiting for a ticket. Give it a workload identity of its own. Inbound auth proves who invoked it, outbound auth is the credential it presents to Kubernetes, cloud APIs, and tickets, and the audit log must name the agent, not the on-call who approved the project.

Amazon Bedrock AgentCore Identity, in AWS’s 2026 developer guide, is built for this shape. It treats agents as workload identities with inbound JWT authorization and outbound credential providers. The documented IdPs include Amazon Cognito, Okta, Microsoft Entra ID, and Auth0.

I am not asking you to buy that product. I am pointing at the mechanism: a distinct agent identity, inbound proof of who called it, and outbound credentials it can use without copying a human’s session.

AgentCore Runtime, in the same guide, assigns distinct identities to agents and supports inbound auth plus outbound flows to third-party services, on behalf of a user or autonomously, with OAuth or API keys. That split matters on an ops desk.

A restart on a named deployment can run as the agent. A spend against a vendor account should still be on behalf of a person, with a token audience bound to that tool.

HTTP MCP uses the same split, with audience-bound tokens, and I cover the spec under MCP authorization below. I keep the identity small: one agent, one job, one audience. The worker that can restart checkout-api does not also hold the token that can open a production change in Terraform.

Name the identity after the job, not after a person, with inbound JWT, audience-bound outbound credentials, an audit trail, and a kill switch that revokes mid-thought.

Allowlist the surface

The allowlist is the world the agent is allowed to see: hosts, APIs, namespaces, commands, ticket queues. If a name is not on the list, the runtime refuses the call before the model gets a retry. Extra tools are how an ops agent wanders from “restart this deploy” into “describe every secret.”

I write the list as data, not as a paragraph in the prompt: checkout-api in prod-eu, the PagerDuty service for that API, the metrics query for that service, GetDeployment, RestartDeployment, GetAlert, SilenceAlert. The list is not the cluster, not AWS, and not a shell.

A shell is an infinite tool, and infinite tools are how demos become incidents.

Amazon Bedrock AgentCore Policy, in AWS’s 2026 developer guide, intercepts agent traffic through AgentCore Gateway and evaluates each request against Cedar policies before tool access. The guide also documents natural-language authoring that compiles into Cedar and flags rules that are overly permissive, overly restrictive, or unsatisfiable.

The mechanism I care about is the intercept: the decision sits outside the agent’s code, so a prompt injection cannot talk the policy away. AgentCore Gateway is the documented place that happens. It converts APIs and Lambda functions into MCP tools, with inbound auth of the caller and outbound auth to the tool.

That is the right shape even if you never touch AWS: one gate, named tools, auth on the way in and on the way out, policy on the call, not in the system prompt.

MCP Security Best Practices in the 2026-07-28 spec name the rest of the list: token passthrough, confused deputy, scope minimization, and SSRF on metadata URLs that point at link-local cloud metadata. I treat those as allowlist bugs: host, audience, scope, and redirect, not only which kubectl verb.

On the list: get and restart this Deployment, describe this autoscaler, ack this ticket, query this recording rule, open a revert PR. Off the list: kube-system and Secrets, IAM and billing, deleting the team, arbitrary PromQL, merge to main, terraform apply.

Read is not write. I put them in different tools with different grants so the model cannot “just this once” upgrade a get into a delete.

Anthropic’s Building effective agents (December 2024) spends an appendix on the agent-computer interface: treat tool descriptions like a docstring for a junior, poka-yoke the arguments, and test how the model actually calls them. For ops, poka-yoke means the restart tool takes a deployment name from an enum, not a free-text kubectl string.

Dry-run every write

A write tool that cannot plan is not ready. Dry-run is the mode where the agent calls the same tool, with the same schema, and the runtime returns the command, the target, and the expected state without changing the world. I keep the agent in that mode until eval is boring, then I promote the grant, not the prompt.

Anthropic’s same 2024 essay says agents should gain ground truth from the environment at each step, from tool results or code execution, and that you should test extensively in sandboxed environments with guardrails. Dry-run is that ground truth without the blast.

The plan is a fact I can grade. “I would restart checkout-api in prod-eu, expected readyReplicas=3, maxUnavailable=0.” If the plan names the wrong namespace, I failed the eval. I did not fail production.

I make dry-run the default in the schema: dry_run true unless a separate grant flips it. Idempotent writes where I can: restart with a client-supplied run id, silence with a ticket id, scale to an absolute count rather than “add two.”

A retry storm that scales twice is how unsupervised agents fail in public. The runtime caps steps and spend. The tool refuses a second mutate with the same run id.

Same name, same schema, same identity: only the mutate bit changes when you promote. The plan includes target, current state, expected state, and the grant that would be used, and the runtime logs it as an event, not a chat line. If a vendor API has no dry-run, you do not have a write tool.

I do not hide dry-run behind a system prompt that says “always think first.” The model will forget, and the runtime will not, if the mutate path is a different grant.

This is the same split I use in How to put an AI agent in production: search is not send, a draft is not the list, a branch is not production. For ops the nouns change: describe is not restart, restart is not rollout, and rollout is not apply.

Eval on frozen incidents

Eval is how I know the dry-run is not theater. I freeze last quarter’s pages, replay them in a sandbox, and grade the plan against what a careful human did or what the postmortem required. Pretty traces that restart the wrong deploy are zeros.

Anthropic’s Demystifying evals for AI agents (9 January 2026) is the operational text I keep on the desk. An eval is an input plus grading logic. A task is one test, a trial is one attempt because the model varies, a transcript is the full tool log, and the outcome is the final state in the environment, not the last sentence.

An ops agent that says “restarted” while readyReplicas is still one has failed, the same way a booking agent fails when it says “booked” and the reservation table is empty.

They split graders into code-based, model-based, and human. For ops I want code first: did the plan name the deployment in the page, stay inside the allowlist, call dry-run before any mutate, and move sandbox replicas only after the write grant was on?

Model graders can score tone in a ticket comment. They should not be the only score. Anthropic is explicit: grade what the agent produced, not the path it took, so you do not punish a valid plan that used a different legal tool order.

They also split capability evals from regression evals. Capability starts hard: a stuck canary, a node drain, a flapping probe. Regression should sit near a full pass on last month’s pages, and as a capability task saturates it graduates into regression.

Start smaller than you think. Anthropic’s 2026 eval piece says 20 to 50 simple tasks from real failures is a start, and that two domain experts should independently reach the same pass or fail. For ops that is a frozen bundle (alert, query, ticket, change, postmortem), and if two SREs disagree on the restart, fix the task before you blame the model.

Run more than one trial. pass@k asks whether any of k attempts worked, and pass^k asks whether all k worked. On-call is pass^k: a restart that works two times out of three is not a Saturday grant.

I log every tool call: name, arguments, latency, error, cost, grant. If I only keep the final message, I will not see the second scale that doubled capacity.

Amazon Bedrock AgentCore Evaluations (2026 developer guide) scores Strands and LangGraph traces via OpenTelemetry, with built-in and custom evaluators. Use any harness that isolates trials, records the transcript, and grades the outcome.

Anthropic warns that leftover state between trials (files, caches, git history) makes the score a lie, so each replay starts from a clean cluster or a snapshot. NIST’s Measure function is this section under another name. If I cannot say the score, I am still in demo.

Then write tools

Write tools come last on purpose. Identity, allowlist, dry-run, and eval have to hold, or a mutate is just a demo with blast radius. When I open a write, I open one class of mutation, never a mandate to fix production.

Promotion is a grant change, not a model change. The tool name stays, dry_run flips to false for that identity in that namespace, and the runtime still caps steps and spend. If the regression set dips, the grant goes back to plan-only.

I watch the first week with a person at the edge, not inside every loop. They set the job and the grants, they read the traces after the run, and they do not click each kubectl.

For ops, autonomy is earned per mutation class. Alert triage can go unsupervised before a rollout can, and a rollout can go before a terraform apply. I have not yet met an apply I would give an unsupervised model on a Friday.

• One write tool per mutation class. RestartDeployment is not RolloutCanary.
• Enums for names, absolute counts for scale, ticket ids for silences, and idempotency keys on every mutate.
• A change window the runtime enforces. The model does not get a vote on freeze.

The researcher that reads logs should not hold RestartDeployment, and the writer that drafts the postmortem should not hold the cloud keys. Extra agents add extra failure, so the default is still one operator with a short list, split only when the keys should never sit together.

Runtime, policy, and isolation

The model should not own the loop. The runtime retries a dropped connection, remembers what already ran, caps spend and steps, enforces the allowlist, and stops. A bad session should not leave files, memory, or credentials for the next one.

Amazon Bedrock AgentCore Runtime (2026 developer guide) hosts agents and tools with session isolation: each user session runs in a dedicated microVM with isolated CPU, memory, and filesystem. After the session, the microVM is terminated and memory is sanitized.

The same page documents microVMs for serverless sessions up to eight hours, and Instances on AWS-managed EC2 for persistent sessions up to fourteen days. I cite those bounds because they are in the guide, not as a reason to skip your own stop rule.

YouTube

Open original

LangChain walked the same runtime-and-policy story when they shipped LangGraph 1.0 on 22 Oct 2025.

You can get the same shape without that product: a container per run, a network policy that only reaches the allowlist, a secret that dies with the task, a hard deadline. If an ops agent runs in production here, it runs on a runtime I can kill.

Policy belongs in that runtime or in the gateway in front of it, not in the prompt. AgentCore Policy’s documented intercept at Gateway is the pattern: every tool call is evaluated before execution, including input parameters and identity. I want that permit in a file, reviewed like any other control, logged like any other deny.

Observability is the transcript plus the outcome: which identity, which grant, which tool, which target, which plan, which mutate, which ticket. If I cannot answer who restarted checkout-api at 18:04, under which grant, I am not in production.

MCP authorization on HTTP

If the ops agent talks to tools over HTTP MCP, authorization is required. The July 2026 MCP authorization spec says the protocol can skip auth, and that stdio should take credentials from the environment instead of OAuth. The moment those tools are on a network, I follow OAuth 2.1, protected resource metadata, and audience-bound tokens.

Clients must send the resource parameter so the token is minted for that server. Servers must validate the audience, refuse anything else, and must not accept or transit other tokens. That is the written ban on token passthrough.

For ops it means the Kubernetes MCP server does not forward a user’s cloud-admin JWT to the cloud MCP server. Each audience gets its own token, and each tool class gets its own scope.

Scope minimization in the MCP security practices is the allowlist, written as OAuth scopes. Start with a small read scope, then step up when a write is actually requested, with a 403 insufficient_scope and a precise challenge, not the whole catalog. Wildcard scopes are how a stolen token becomes a stolen shift.

The same practices cover confused deputy, SSRF on metadata URLs including link-local cloud metadata, session ids that must not double as authentication, and local MCP servers that are binaries on the box. A laptop Kubernetes MCP server with the user’s kubeconfig is a human identity with extra tools. Production is a remote server with its own identity and no default path to the local kubeconfig.

Tool descriptions are untrusted unless they come from a server you trust. The MCP spec’s trust and safety notes say tools are arbitrary code execution, and that annotations about behavior should be treated with caution. I pin the tool list, and if the schema changes the eval runs again with the grant on dry-run until it passes.

Keep a runbook a runbook when you can draw the path

Most ops work is a workflow, with a known path and known checks. Anthropic’s Building effective agents (December 2024) draws the line: workflows follow predefined code paths, and agents direct their own tool use. Use the workflow when you can draw the path.

A runbook that always restarts checkout-api after a failed readiness probe, then waits for readyReplicas, then comments on the ticket, should stay a runbook. How to auto-run ops runbooks is that job. Putting an open agent on it adds latency, cost, and a chance it will “help” by scaling a different service.

Anthropic’s advice is to start with the simplest thing that works, and to add agentic loops only when they demonstrably improve outcomes. A script with no model is allowed to win.

The agent earns its keep when the path is not drawable in advance: a novel error budget burn across two regions, a canary that is green on metrics and red on a subset of users, a dependency that is failing in a way the runbook never named. Even then I keep the tools short and the finish line testable: a plan, a ticket, a single allowed mutate.

Anthropic lists three implementation principles I steal for ops: keep the design simple, show the planning steps, and spend real effort on the tool interface. Transparency for me is the dry-run plan in the ticket, not a chain-of-thought dump in Slack. The plan is the artifact a human can read at the edge, and the mutate is what the runtime does after the grant.

Probe-fail-restart-wait stays a runbook, and silence-this-alert is one tool, maybe no model. Novel burn across two regions is an agent on dry-run. “Fix production” is neither, because there is no finish line.

What usually breaks

Ops agents fail in boring ways, and the model is rarely the first cause: the grants are. I keep a short list on the wall because every one of these has shown up in a trace I later had to read in public.

• A human’s cloud key in the runtime. The audit log names the wrong actor.
• A shell tool. The allowlist was a suggestion.
• Write tools on day one, eval after the first bad restart.
• Dry-run in the prompt only. The mutate path was always live.
• One agent with restart, apply, and payout in the same body.
• MCP over HTTP with no audience-bound token. The gateway forwarded whatever it got.
• An eval of chat quality. The replica count never moved in the sandbox.
• No spend or step cap. A retry storm was the first invoice and the first incident.
• A runbook replaced by an agent because the slide said agentic. The path was already drawn.

When it breaks, I do not start with a bigger model. I start with the transcript and the outcome. Anthropic’s 2026 eval essay is right that you will not know if graders work unless you read the traces, and failures should look fair: it is clear what the agent got wrong.

If the task was ambiguous, I fix the task. If the grant was too wide, I close it. If the runtime could not stop, I fix the runtime rather than patch the prompt.

When you can call it production

It is production when the five grants hold on a desk that can page you. The agent has a name in the IdP, the allowlist is data, dry-run is the default, the frozen set beats the baseline, and one write tool is live for a boring week. Then you may open the next write.

It is not production when a person must approve every tool call. That is a slow chatbot with kubectl. People stay at the edge (the job, the grants, the weekly score, the change window) and step into a trace when the score fails.

How to put an AI agent in production is still the outer frame, and this guide is the ops-shaped inside of it. A tight operator can ship the dry-run path in days. The slow part is frozen pages, policy, identity, and reading failures, and skipping that makes the first users the production cluster.