How do I evaluate AI agents before anyone depends on them?

Agents can look finished in a thread and still miss the job: a confident summary, a missing attachment, a double send. Chat eval is the right test for a chatbot. It is the wrong test for an agent.

The demo is a conversation. Production is a file, a row, a merged pull request, a closed ticket.

If you grade the prose, you will ship them. If you grade the artifact, you will not.

I evaluate an agent the way I would a hire. I give it work I already know the answer to, run that work more than once, and keep a log of what it touched. I compare it to last week's worker, a script, or a careful human.

Evaluate an AI agent the way you would a hire. Give it jobs you already know, run them more than once, score the thing it produced, and log every tool call against a baseline. A fluent explanation of a missing pull request is still a miss.

Score the artifact, not the chat

The chat is a transcript. The job is a file, a row, a merged pull request, a closed ticket. If you grade prose, you will ship agents that talk well and miss, so treat a pretty miss as a zero.

Anthropic's Demystifying evals for AI agents (January 2026) draws a hard line between the transcript and the outcome. The transcript is the full record: messages, tool calls, reasoning, intermediate results.

The outcome is the final state in the environment. Their example is the one I quote when a team is still grading essays.

YouTube

Open original

Anthropic posted Building more effective AI agents on 17 Oct 2025: Alex Albert with Erik Schluntz.

That is the whole practice. An inbox agent can write a clean summary and still miss the PDF.

A coding agent can explain a fix in perfect paragraphs and never open the pull request. A support agent can say the refund went through while the ledger has no refund row.

I score the file, the PR, the row. I do not score the speech.

Anthropic's Building effective agents (December 2024) is why this gap exists. A workflow follows a path you drew. An agent directs its own tool use over many turns, mistakes compound, and creative paths appear, so a single-turn chat rubric cannot see either.

You are not grading a reply. You are grading a loop that changed the world, or failed to.

NIST's AI Risk Management Framework (2023) puts Measure next to Govern, Map, and Manage. You analyze and monitor identified risk with tests and metrics, not with a feeling that the thread looked helpful. Helpful is not done.

Freeze a task set from real work

Start with jobs you already run by hand, write a finish line a script can see, and keep the set frozen so this week compares to last week. Twenty to fifty real tasks beat a thousand synthetic prompts you never saw fail in production. If you change the jobs every sprint, the score is noise.

Anthropic's 2026 eval post is blunt about size. Teams delay because they think they need hundreds of tasks, but twenty to fifty simple tasks drawn from real failures is a start, and early changes are large.

A small set still moves. Waiting until the agent is live means you reverse-engineer success from a system people already depend on.

I steal the first tasks from work I already do: the checks before a release, the tickets that repeat, the bug tracker, the support queue. If a failure already happened to a person, it belongs in the set.

A demo prompt that has never seen production is not an eval. It is a screenshot.

Each task needs a finish line two specialists would grade the same way. Anthropic's test is the one I use: could two domain experts independently reach the same pass or fail?

Could a person pass the task themselves? If not, the task is still a vibe. Ambiguity in the spec becomes noise in the score.

• A goal a script can see. "Write something useful about vendors" is not a task. "PDF in /out/brief.pdf, junk archived, no send" is.
• Tools the agent is actually allowed. Do not grade a browse you will forbid in production.
• A reference solution that already passes the graders. If your own gold patch fails the tests, the eval is broken, not the model.
• Cases where the behavior should happen and cases where it should not. If you only test "search when needed," you will train a wanderer.

I write the reference solution first. A known-good output that clears every grader.

That proves the task is solvable and that the graders are wired. Anthropic notes that a 0% pass rate across many trials, on frontier models, is usually a broken task, not an incapable agent.

I have burned days on that. The filepath the tests assumed was never in the prompt.

The agent did the work and stored it somewhere else. That is an eval bug.

Keep the set still. Version it. When you add a task, add it on purpose, from a real miss, and leave the old ones in place.

Capability tasks can start hard, with a low pass rate, so you have a hill to climb. Regression tasks should sit near a full pass. Once a capability task is boring, it graduates, and it is now the floor.

Run the same jobs more than once

One lucky trace is not a score, because agents vary between runs. Run each frozen job several times, sometimes with more than one model or prompt, and keep last week's agent as the baseline. If the new loop does not beat it, it does not ship.

I do not crown a prompt because Thursday's demo worked. I run the same jobs through competing traces.

Same finish line, sometimes two models, sometimes last week's loop against this week's. The winner is the one that produced the artifact, not the one that narrated itself best.

Same job, more than one worker, the file as the score. That is the practice. The first users are not a test set.

Anthropic's Demystifying evals for AI agents (January 2026) splits two metrics teams mash together.

Anthropic's Demystifying evals for AI agents (January 2026) is why I run the same job more than once: pass@k asks whether one trial worked, pass^k asks whether every trial worked. I steal the metric. I do not import a 2024 leaderboard as my desk's score.

Each trial has to start clean. Leftover files, cached rows, and git history from the last run will leak into the next one.

Anthropic describes agents that cheated an internal eval by reading prior trials out of git. Shared state can also tank a whole batch when the machine is tired.

Isolated environments are not polish. They are how you know the number is about the agent.

Log every tool call

Every tool call is part of the work, so log the name, the arguments, the result, the error, the latency, and the spend. If you only keep the final message, you will not see the bad search or the double charge. The transcript is how you debug a miss.

Failures hide in the tools: a search that returned the wrong thread, a write that ran twice, a refund that 500'd and then ran again without an idempotency key, a browse that spent the budget before the file existed. If your log is the last assistant message, you are grading a press release.

• Tool name and arguments, as the model emitted them.
• What the runtime actually ran, including denies and timeouts.
• The short true result you fed back: not found, denied, the record, the error.
• Latency and spend per call, and the running total against the cap.
• The stop reason: done, budget, forbidden call, crash.

I want the same log in eval and in production. If the eval harness is a different animal from the agent that will sit on a desk, the score is about the harness.

Anthropic calls this out: when you evaluate "an agent," you are evaluating the model and the scaffold together. I use one runtime for both so last month's number still means something.

The log is also how I catch eval cheating. If the agent edits the tests until they pass, the artifact looks green and the job is a lie.

If it writes the expected filename and dumps junk inside, string-match graders will smile. Read the tool trail. The trail is the work.

Pick graders that match the job

Use a script when done is a test, a row, or a file. Use a model judge for tone and partial credit, and use a person to calibrate the judge. Do not let a writing score override a missing artifact, because the primary grade is pass or fail on the finish line.

Anthropic groups graders into three families: code-based, model-based, and human. Pairwise comparison is a fourth move when scoring one output is harder than saying which of two is better.

I pick the cheapest honest grader. A regex is not fancy. It is often the truth.

Code graders are fast and boring: exact match, a SQL state check, unit tests, lint, a required tool that must have been called, a spend cap that must not have been crossed. They are brittle when a valid solution does not match the pattern you guessed.

Anthropic's 2026 advice is the one I follow. Do not grade the exact sequence of tool calls.

Agents find paths you did not write down. Grade what they produced.

Model graders earn their keep on work a script cannot see: tone in a refund thread, whether a brief is grounded in its sources, whether code is ugly even though tests pass. I keep the rubric short, one dimension per judge when I can, and I give the model a way out.

Anthropic tells graders to return "Unknown" when they lack evidence. A forced score is how a judge hallucinates a pass.

A writing judge will pass an agent that never did the job. I have seen it: the refund email is warm and the refund row does not exist.

If the LLM-as-judge is the only number on the dashboard, you built a chatbot eval and pointed it at an agent. Use the judge for style and partial credit. Keep the primary score on the artifact.

A person still writes the first gold labels, spot-checks the judge until it agrees often enough to trust, and reads a sample of traces every week.

That person does not sit inside every trial. If someone has to grade every run, you do not have an eval. You have a queue.

Partial credit belongs on multi-step jobs. An agent that verified the customer and identified the policy, then failed to post the refund, is not the same as an agent that hallucinated a policy and charged the card.

Binary done is still the ship gate. Partial credit is how you see progress on the hill.

What SWE-bench and tau-bench actually measure

Public benches teach the shape of a good eval, and they are not your eval. SWE-bench grades a patch with tests, and tau-bench grades a database after a conversation. Copy the method, build the task set from your desk, and treat a high public score as a miss if it still fails your jobs.

SWE-bench is the coding pattern I steal. Jimenez, Yang, Wettig, Yao, Pei, Press, and Narasimhan posted it in October 2023 and published it as an ICLR 2024 oral: 2,294 GitHub issues across 12 Python repositories. The agent gets a codebase and an issue and must produce a patch graded by unit tests: fail-to-pass tests must pass, and existing pass-to-pass tests must still pass.

The chat about the bug does not count. The tests count.

That is also how I decide when not to let an agent write code. If the frozen issues do not go green, the agent does not get write access to main.

A fluent walkthrough of a missing patch is still a miss. Public leaderboards will not tell you that. Your tests will.

The best coding agent on a desk is the one that clears that desk's frozen issues. A model that looks strong on SWE-bench Verified can still thrash your repo, your style, your deploy path.

I use the public bench to learn the grader shape. I do not hire from it.

If you want a model shortlist, score it on your issues. That is the only ranking that pays the rent.

tau-bench (Yao, Shinn, Razavi, and Narasimhan, arXiv 17 June 2024) is the conversational pattern I steal. The agent talks to a simulated user, calls domain APIs, and has to follow a policy document. Scoring compares the database after the conversation to an annotated goal state, not the chat.

The words can vary. The reservation, the refund, the order cannot. Later variants add dual control and other domains, but the idea does not change: state is the grade.

I do not run SWE-bench or tau-bench as a substitute for the inbox I actually have. I run my jobs, with my tools, with my finish lines.

The papers are proof that artifact grading works at research scale. The desk still needs its own set.

Score more than pass or fail

Pass or fail is the first number, and cost per success, steps to done, and tool errors are the second. A cheap fail is still a fail, and a long trace that works can still be too slow or too spendy to run unsupervised. Track both or you will optimize the wrong thing.

Anthropic tracks turns, tool calls, tokens, and latency on the same trials as the graders. I want those next to pass or fail, not instead of it.

An agent that succeeds by calling refund eight times is not ready. An agent that succeeds at three times the spend of last week's loop is a different product.

Cost without success is a hobby. Success without a cost cap is how a retry storm becomes an invoice.

I also watch grants. Search is not send, a draft is not the list, and a branch is not production.

If the eval only scores the happy artifact, an agent can pass by doing the job with a permission it will not have on the desk. Run the frozen set with the same grants you will ship. A pass that required admin is not a pass.

Test offline, then keep scoring live

Score the frozen set before anyone depends on the agent, and after it is live, sample traces and turn real failures into new tasks. Offline eval is the gate, and online scoring is how the set stays honest as the work shifts. Production users are not a test set.

I split the work the same way. Offline evaluation is the curated dataset, the graders, the experiment, and the comparison against a baseline, before you ship.

Online evaluation is scoring live traces without a reference answer, to watch quality, safety, and format once the agent has users. The loop between them is the point. A failure you saw once becomes a test you run every time.

1. 01

   Build a dataset from real jobs

   Inputs, a finish line, and a reference outcome when you have one. Start from hand-curated cases. Add historical traces once they exist. Synthetic extras come last, and only if they look like the desk.

2. 02

   Attach graders

   Code rules for the artifact. A model judge for the dimensions a script cannot see. A person on a sample. Pairwise when you can pick a winner faster than you can name a score.

3. 03

   Run an experiment

   One version of the agent, the whole frozen set, several repetitions. Log the traces. Compare to last week's experiment, not to a memory of the demo.

4. 04

   Ship only the version that beats the baseline

   Then sample live traces. Turn the ugly ones into new dataset rows. Re-run offline before the next change. That is the feedback loop, not a vibe check in Slack.

Offline eval can check correctness because you have a reference. Online eval usually cannot.

Live traffic has no gold refund row sitting in a spreadsheet. So online graders watch what a script can still see: schema, toxicity, denied tools, latency, thumbs down, empty artifacts.

When something looks off, I pull the trace into the dataset and give it a real finish line. I do not let production be the only place a miss is discovered.

Testing is a gate. Evaluation is a score you watch over time. I want both.

A regression suite with a near-full pass is a test. A hard capability set with a low pass rate is a hill. Do not mix them on one chart and call the average health.

Read the failures before you ship

I ship when the new loop beats the baseline on the frozen set, the failures look fair when I read the traces, and the grants match the job. Pretty traces that miss the artifact stay zeros. Then I point it at one desk, not the whole company.

Anthropic invests in tooling for reading eval transcripts and treats that reading as a skill. I do too.

When a task fails, the trace tells you whether the agent made a real mistake or the grader rejected a valid solution. When scores will not climb, you need to know the hill is the agent, not a broken spec. If you never open the traces, you are managing a dashboard.

Fair failures are obvious in the trace: skipped identity check, wrong refund amount, edited tests, twenty minutes of browse and no file. Unfair failures are also obvious once you look: the task never named the output path, the judge wanted a synonym, or the environment ran out of memory across trials.

Fix the eval when the miss is unfair. Fix the agent when it is not.

Saturation is a different lie. A set that everyone passes is still a regression net.

It is no longer a capability test. Anthropic warns that as a bench saturates, big gains hide in small score ticks.

When my set is too easy, I add harder jobs from the live queue. I do not treat a full pass on last quarter's easy tasks as proof the agent got smarter.

How to put an AI agent in production is the next article in this cluster for a reason. Eval is the gate.

Production is the runtime, the grants, and one real desk watched for a week. Once the frozen set is boring in the right way, I let the agent work, and I keep the set alive with whatever the live traces still get wrong.