When should I not let an AI agent write or merge code?

I let agents open pull requests, and I do not let them merge. Write is a draft on a branch the agent created. Merge is a decision about secrets, data, identity, and the pipeline that can read both.

Teams collapse those two acts into one sentence: the agent writes code. Write is a patch. Merge is production, and CI is a privileged interpreter sitting between them, so an agent that can edit a workflow file can ask that interpreter to print the vault.

This is the stop list. The companion runbook is how to run a coding agent in your repo. Here I teach the gates I refuse to put in a prompt and hope for.

An agent can write a patch. It should not merge, mint secrets, run a migration, change auth, or ship unsigned CI. Those are gates, not vibes, and I treat them as runtime denies and required reviews, the same way a junior hire does not get production keys on day one.

The gate is merge, not write

Write means a branch the agent owns and a pull request someone else can still reject. Merge means the default branch, the deploy, and every secret the pipeline can see. I let the first happen often, and I never let the model be the person who clicks the second.

GitHub’s Copilot coding agent docs (2025, still current in 2026) say it can only push to branches it created, usually under a `copilot/` prefix. It cannot approve or merge its own pull request, and it cannot mark that pull request ready for review. The developer who assigned the issue cannot satisfy a required-approvals rule by reviewing that same pull request, so another person has to look.

That split is the product, and it is the policy I copy when I wire my own coding loop on Kernel. The model may call write_file and open_pr. It does not get merge_pr or a deploy tool, because opening a pull request is not production.

r/LocalLLaMA was still arguing the same split on 6 Jun 2026: a self-hosted coding agent still needed a human at merge.

If your branch protection is off, you do not have a coding agent. You have a model with a push key. Turn required reviews on before you turn the agent on, and make the required check a human who did not prompt the run, because if the same person assigns the issue and clicks approve, the gate is theater.

I still read the diff. An agent that cannot merge can still put a bad idea on a branch, and a tired reviewer can still click the green button. The runtime removes the merge tool so a bad night cannot skip the queue; review remains a job, and the gate only makes that job possible.

• Agent tools: read, search, write on an allowlisted tree, open a pull request, reply on the thread.
• Human tools: approve, request changes, merge, deploy, rotate a secret, run a migration.
• Never on the agent: merge, production deploy, vault write, workflow edit, auth path edit.

Secrets do not belong in the prompt

A secret that enters the model context is already gone. Logs, traces, eval dumps, and the next prompt cache will keep a copy, so I never hand the agent a raw token. I hand it a named tool that the runtime can call with a credential the model never sees.

This is AI agent tool calling with the dangerous half deleted. search_mail is not send_mail, and read_config is not print_env. A tool named get_database_url that returns the string into the transcript is a leak with an API; a tool named run_readonly_query that executes inside the sandbox and returns a row count is a job.

I deny the agent any path that is a secret by convention: .env, .env.*, **/*secret*, **/credentials*, vault YAML, Terraform tfvars with keys, Kubernetes Secret manifests, 1Password or AWS parameter files committed by mistake. CODEOWNERS on those globs is not optional, and a secret scanner as a required check is not optional. The runtime also redacts token-shaped strings before a tool result goes back to the model.

The failure I actually see is quieter than a committed AWS key. The agent writes a debug log that prints process.env, adds a fixture that hard-codes a staging password so a test can boot, or pastes a webhook secret into the pull request body so a reviewer can try it. All three are secret writes, and all three should fail CI and fail the tool allowlist.

NIST's AI Risk Management Framework 1.0, published in January 2023 as NIST AI 100-1, lists privacy-enhanced and secure and resilient as trustworthiness characteristics, with valid and reliable as the base. A coding agent that can echo a vault value is not privacy-enhanced. It is also not valid for the job you thought you assigned, which was fix the bug, not hold the keys.

If the job needs a secret, the runtime injects it as an environment variable inside a sandbox the model cannot read back. The tool result the model sees is applied, denied, or a redacted error. I log the tool name, the duration, and the outcome, and I do not log the payload.

• Keep secret-shaped strings out of prompts, pull request bodies, and eval traces.
• No write tools on .env, vault files, or cloud key material.
• Named tools for the work, with credentials injected outside the model.
• Redaction on the way back in, a scanner on the way out, and rotation by a human if a leak is even possible.

Migrations run once

A migration is not a refactor. It is a one-way change to the store every future deploy will inherit. An agent may draft additive SQL on a branch, and it does not run migrate, DROP, or squash history to make the diff look small.

I use expand and contract. Expand adds a column, a table, or a backfill that can sit next to the old shape; contract removes the old shape after the readers have moved. The agent may propose the expand file, and a person who owns the data signs the contract step and runs it.

CI fails the pull request if the migration file contains DROP, TRUNCATE, DELETE FROM without a WHERE, ALTER DROP COLUMN, or a rewrite of an old file in place. Those patterns are not always wrong. They are always too large for a model to ship unsupervised, so a human can add a label that acknowledges the blast after reading the plan.

Put CODEOWNERS on `db/migrations`, `prisma/migrations`, `supabase/migrations`, and whatever folder you actually use. The agent has `write_file` on application code and it does not have a migrate tool. If I need a dry run, I give it `explain_sql`, which returns the plan, not `apply_sql`.

Data shape is also product. Renaming a column that billing still reads will pass unit tests and break the invoice job, and I do not ask the agent to be a historian of every reader. If the ticket is "add an index", the patch is one file and a note about lock time; if the ticket is "move users to a new table", that is a project with a human owner, not an agent loop.

Backfills deserve the same gate. A script that rewrites every row is a migration with a longer runtime, so the agent may draft it and may not point it at production. The runtime has no production database tool at all; staging gets a read-only tool until a person promotes the job, the same way I promote any other unsupervised task after eval.

Auth is a human surface

Auth is the code that decides who someone is and what they may touch: session, JWT, cookies, OAuth, SSO maps, RBAC tables, password hashing, CSRF, CORS, public route lists. I do not let an agent patch those paths. A one-line change to a cookie flag is a company incident with a green test suite.

The agent may write tests against the current behavior, document the current flow, and add a feature behind middleware that already exists, without editing the middleware. It may not add a bypass, widen a role, turn off secure or httpOnly, or skip verification so a demo boots.

I enforce this as a path deny in the runtime, not as a paragraph in the system prompt, because prompts are suggestions and tool schemas are law. The write tool rejects files under auth/, identity/, middleware/session, and the files CODEOWNERS already assigns to a security owner. If the model argues, the runtime returns denied.

Public route tables and CORS origins belong on the same list, along with webhook signature checks and the code that trusts a header from a proxy. Agents are good at making a request succeed; auth is the code that must sometimes make a request fail. Those two instincts fight, so I do not put them in one loop.

If a ticket is "users cannot log in", I still start with logs and a failing test, which an agent can draft. The patch that changes the verifier is mine, or it is the security owner's. Anthropic's Building effective agents, published 19 December 2024, says agents can pause for human judgment at checkpoints, and auth is that checkpoint every time, not when the model feels unsure.

Unsigned CI is unsigned code

A workflow file is code that runs with secrets. If an agent can edit .github/workflows, GitLab CI, or the deploy YAML, it can add a step that prints those secrets to an endpoint it controls. That is the fork-PR attack, now with a model that will write the YAML on request.

GitHub's own Copilot coding agent docs, current through 2026, treat the agent as an outside collaborator. Actions workflows do not run when Copilot pushes until a person with write access clicks Approve and run workflows. The docs tell you to inspect the pull request first, and to be especially alert to changes under .github/workflows/; that sentence is the policy, and I do not skip it.

In March 2026 GitHub added a repository setting so admins can skip that human approval and let workflows run immediately on Copilot pull requests. The changelog is explicit: the default still requires a human, because workflows may have tokens, secrets, or repository permissions. I leave the default on, because speed is not a reason to run untrusted YAML against the vault.

Unsigned here means two things, and I gate both. The workflow run is unsigned until a human approves it, and the commit itself may not meet a signed-commit ruleset. An agent that cannot produce a GPG or SSH-signed commit should fail a require signed commits check; that is a working gate, not a defect to disable so the bot feels productive.

CODEOWNERS on .github/, .gitlab-ci.yml, buildkite/, and any Terraform that binds OIDC to a deploy role. The agent has no write tool on those paths. If a workflow must change, a person writes it, or a person copies a generated draft into a second pull request they own, because the agent does not get to be the author of the file that holds GITHUB_TOKEN and the cloud role.

I review workflow diffs as if they came from a fork. New uses of pull_request_target, expanded permissions blocks, extra pull-requests: write, a curl to an unknown host, a change to environment protection rules: any of those and the answer is no, even if the rest of the feature looks right. The feature can wait; the vault cannot be un-printed.

• Keep Copilot and any other coding agent as an outside collaborator for Actions.
• Do not enable skip-approval so the tests feel faster.
• Path-deny workflow files, require signed commits, and read every YAML diff as a privilege change.

What GitHub already requires

I do not invent this split. GitHub documented it when Copilot coding agent shipped in May 2025, and the docs still hold. The agent works in a sandbox with a firewall, read-only access until it pushes its own branch, and existing branch protections still apply.

The default policies, from GitHub's 2025 product post and the current docs, are the ones I copy into Kernel even when the model is not Copilot: push only to agent-created branches, and the person who asked for the pull request cannot be the required approver. Internet is an allowlist, not the open net. Actions stay held until a human inspects the diff, and the agent cannot merge.

That paragraph is about Copilot, and it is true of every coding agent I run. A local loop on a laptop is worse, because there is no outside-collaborator flag, and if your agent uses your shell, it uses your credentials. I do not give a local agent the production kube context, an org-wide GITHUB_TOKEN, or a push to main.

GitHub also notes that Copilot's own review comments do not count as approvals and do not block merge. An agent reviewing another agent's pull request is not a two-person rule. Two models in a circle will agree, so the required reviewer is a person with a name on the CODEOWNERS file.

If you skip these defaults to move faster, you are not running Copilot as designed. You are running an untrusted contributor with Actions secrets.

The March 2026 skip-approval setting exists for teams that accept that risk. I do not accept it on a repo that holds customer data, billable events, or identity.

Start with a workflow

Most of the jobs people want a coding agent for are already a path you can draw: open the ticket, write the failing test, patch the file, run the suite, open the pull request. That is a workflow. Anthropic's Building effective agents, 19 December 2024, draws the line I use.

They recommend finding the simplest solution possible, and only increasing complexity when needed. For many applications, a single LLM call with retrieval is enough. When you need more, workflows give predictability, while agents are for open-ended work where you cannot hardcode the path and where you can still trust the decision-making enough to let it loop.

A coding agent that may touch secrets, migrations, auth, or CI is not a trusted environment, so I do not start with an agent. I start with a workflow that has gates: the model may draft, a script checks the diff against the deny list, tests run, a human reviews, merge is a person. Prompt chaining with a gate on each step is the pattern Anthropic names, and I use it.

Anthropic also writes that coding agents work well because tests can verify behavior, and that human review remains crucial for alignment with broader system requirements. Tests do not know that you promised a customer the old column would stay until June, and they do not know that a workflow permission is a privilege. The human is the system-requirements check the suite cannot see.

If the path is drawable, keep it a workflow. If the path is not drawable, still do not hand over the four surfaces. Give the agent a smaller map: application code under an allowlist, a test command, a pull request tool, which is how to run a coding agent in your repo without letting it become the security team.

Map the blast radius

NIST AI RMF 1.0 organizes the work as Govern, Map, Measure, and Manage. I use those four words as a desk check before an agent touches a repo. They are voluntary in the document and mandatory in mine, because an unmapped agent is a tool with no job.

Govern is the policy in this piece: who may merge, which paths are denied, who owns secrets, who owns auth, who owns the pipeline. If that is not written down, the model will improvise, and improvisation on identity is an incident.

Map is the blast radius: which files this ticket can touch, which secrets CI injects for those files, which environments a bad workflow could reach, which tables a bad migration would lock. If I cannot name those, I do not assign the ticket to an agent. I assign it to a person, or I split it until the map is small.

Measure is eval. I score coding agents the way I score every other operator in how to evaluate AI agents: freeze a task set, run competing traces, log every tool call, score the artifact, keep a baseline. A pretty explanation of a missing test is a zero, and a patch that edits .github/workflows when the ticket was a button color is a zero and a revoked grant.

Manage is what you do when the score drops or the map was wrong: kill the trace, rotate if a secret might have entered the context, tighten the tool list, put the job back behind a workflow. NIST treats valid and reliable as the base of trustworthiness. An agent that cannot stay inside the map is not valid for production, no matter how fast the diffs arrive.

I write the map into the ticket before the agent starts: allow src/ui/button.tsx and its test, deny everything else, and keep CI, env, and migrate off the list. The runtime reads that allowlist, and the model does not get a vote. If the work needs a file off the list, the run stops and a person widens the map on purpose.

The policy I actually run

A policy that lives in a wiki is a prompt. A policy that lives in CODEOWNERS, branch rules, tool schemas, and required checks is a gate. I write both, but I only trust the second, and here is the desk version I keep on Kernel.

1. 01

   Name the four hard surfaces

   Secrets, migrations, auth, CI. Write the globs. Put them in CODEOWNERS with a named owner. Put the same globs in the agent's write-tool deny list. If the two lists drift, the deny list wins.

2. 02

   Give the agent a branch, not main

   The runtime creates the branch. The agent can push only there. Required reviews on the default branch. The assigner is not the approver. Merge and deploy stay human tools.

3. 03

   Hold CI until a person inspects the diff

   Treat every coding agent as an outside collaborator. Do not skip GitHub's Approve and run workflows step. Read .github/ diffs as privilege changes. Signed commits stay required.

4. 04

   Inject secrets outside the model

   No tokens in prompts. Named tools, credentials in the sandbox, redacted results, a scanner on the pull request. Rotate if a secret might have landed in a trace.

5. 05

   Score frozen jobs before you widen the map

   Ten tickets that look like production, then fifty. Log tool calls. A patch that crosses a deny glob is a fail even if the feature works. Promote one surface at a time.

I keep this table next to the tool schema. When someone asks for a new grant, we edit the table first. If the new grant is merge, a secret-write, or a workflow edit, the answer is no, because the model is not the variable; the surface is.

Autonomy is a grant, like a permission. Inbox triage can go unsupervised after it beats a baseline, and a coding agent can open pull requests on UI files after it beats a baseline. Production migrate does not get that grant, because a passing eval on twenty additive indexes does not tell you what happens on the twenty-first DROP.

What I still let an agent write

The stop list is not a ban on coding agents. It is the edge of the map. Inside the map I let agents work, and I let them work a lot, because the gates make the work reviewable instead of scary.

I let an agent add tests, types, docs, and small refactors in modules that already have coverage, and I let it fix bugs where a failing test already exists, or where it can write that test first. I let it build UI behind existing auth middleware, draft the expand half of a migration for a person to read, and open the pull request on its own branch.

Merge, secrets, migrate, auth, and CI stay off that list. The person who prompted the run is never the only required reviewer, and I do not skip the Actions approval so the green checks arrive faster.

If you want the runbook for the work inside the map, read how to run a coding agent in your repo. If you want the score, read how to evaluate AI agents. If you want the contract between the model and the filesystem, read AI agent tool calling. This page is the fence around those three.

A coding agent is useful when the blast radius is small and the finish line is a passing suite plus a human merge. It is a liability when the blast radius is the vault, the database, the identity layer, or the pipeline that can speak for all three.