Skip to main content
    Kriv AI

    Agentic automation

    What is AI agent evaluation? How teams test an agent before release

    Five vendors document how to test an agent. This page puts their common ideas in plain words, with one worked example of scoring a single agent run.

    AI agent evaluation is testing an agent on the work it is meant to do, and scoring both what it produced and how it got there. Anthropic defines an eval as "a test for an AI system." Microsoft says to evaluate "not just the final output, but also the quality and efficiency of each step in the workflow." Source: Anthropic, Microsoft, read 17 November 2026.

    plain answer

    AI agent evaluation tests a whole run, not one answer

    A single model answer is one output. An agent takes several steps: it reads a request, picks tools, calls them, and uses what comes back. Microsoft's Foundry page says an agent workflow "typically involves reasoning through user intents, calling relevant tools, and using tool results to complete tasks like updating a database or drafting a report."

    For that reason Microsoft says that to build production-ready agentic applications "you need to evaluate not just the final output, but also the quality and efficiency of each step in the workflow." Anthropic's wording for the unit being tested is a task, with each attempt at it called a trial. Our page on the LLM evaluation framework covers the model side of testing.

    plain answer

    The words evaluation guides use: task, trial, grader, transcript, trace

    Anthropic's guide defines the terms most other guides assume. A task is "a single test with defined inputs and success criteria." "Each attempt at a task is a trial." A grader is "logic that scores some aspect of the agent's performance."

    Anthropic calls the record of a trial a transcript, also called a trace or trajectory, and says "The outcome is the final state in the environment at the end of the trial." Its example: a flight-booking agent might say "Your flight has been booked" at the end of the transcript, but the outcome is whether a reservation exists in the environment's SQL database.

    OpenAI and AWS use the word trace in the same way. OpenAI: "A trace captures the end-to-end record of model calls, tool calls, guardrails, and handoffs for one run." AWS adds two neighbours: a session is "a logical grouping of related interactions from a single user or workflow," and a tool call is a span that represents "an agent's invocation of an external function, API, or capability."

    plain answer

    Teams score the final result and the path taken

    Google describes the same split in two lines. "Final response evaluation: Evaluate the final output of an agent (whether or not the agent achieved its goal)." "Trajectory evaluation: Evaluate the path (sequence of tool calls) the agent took to reach the final response."

    Microsoft names the two ideas system evaluation, "to examine the end-to-end outcomes of the agentic system", and process evaluation, "to verify the step-by-step execution to achieve the outcomes." Its Foundry page lists system checks such as Task Completion (preview) and Task Adherence (preview), and process checks such as Tool Call Accuracy and Tool Selection. Microsoft says preview items are provided "without a service-level agreement" and that it does not recommend them "for production workloads."

    Both vendors let you choose how strictly the path is matched. Google lists an exact match (the predicted trajectory is identical to the reference), an in-order match (all reference tool calls appear in the same order, extra calls allowed) and an any-order match (all reference tool calls appear, order does not matter). Microsoft's Task Navigation Efficiency evaluator offers the same three modes, named exact_match, in_order_match and any_order_match.

    Anthropic cautions against over-checking the path. It says checking that an agent followed specific steps in a specific order was "too rigid and results in overly brittle tests, as agents regularly find valid approaches that eval designers didn't anticipate," and that it is "often better to grade what the agent produced, not the path it took."

    plain answer

    Three kinds of grader score an agent's work

    Anthropic says "Agent evaluations typically combine three types of graders: code-based, model-based, and human." It lists code-based graders as "Fast," "Cheap," "Objective," and "Reproducible," with the weakness that they are "Brittle to valid variations that don't match expected patterns exactly."

    Model-based graders are listed as "Flexible," "Scalable," and as capturing nuance, but also "Non-deterministic" and "Requires calibration with human graders for accuracy." Human graders are listed as "Gold standard quality" and also "Expensive" and "Slow." AWS describes the model-based kind as a large language model that is prompted with evaluation criteria and "produces a score, label, or explanation."

    plain answer

    A worked example: scoring one refund run four ways

    Anthropic gives an example of a support task where the agent must handle a refund for a frustrated customer. Its example scores the single run in four ways.

    A model-based grader reads the transcript against a rubric with three assertions, including "Agent showed empathy for customer's frustration" and "Agent's response grounded in fetch_policy tool results." A state check confirms the ticket status is resolved and the refund status is processed. A tool-call check requires three tools: verify_identity, process_refund with an amount of at most 100 (written in the example as "<=100"), and send_confirmation. A transcript check caps the conversation at 10 turns.

    The same example also tracks how many turns, tool calls and tokens the run used, and latency measures such as time to first token. Anthropic titles both examples theoretical and says that in practice, coding evaluations typically rely on unit tests and an LLM rubric, with additional graders and metrics added only as needed.

    plain answer

    What to put in a first eval set

    Anthropic's advice is to start small and real: "20-50 simple tasks drawn from real failures is a great start," and "Begin with the manual checks you run during development." A good task is "one where two domain experts would independently reach the same pass/fail verdict." For each task it suggests a reference solution, "a known working output that passes all graders."

    Its general eval guide adds that evals should "mirror your real-world task distribution" and asks you not to forget edge cases. Anthropic also says "You won't know if your graders are working well unless you read the transcripts and grades from many trials."

    plain answer

    Capability evals and regression evals ask different questions

    Anthropic separates two uses. Capability evals ask, "What can this agent do well?" and "should start at a low pass rate." Regression evals ask, "Does the agent still handle all the tasks it used to?" and "should have a nearly 100% pass rate."

    Because an agent can give a different result on each attempt, Anthropic defines two scores over k trials: pass@k "measures the likelihood that an agent gets at least one correct solution in k attempts," and pass^k "measures the probability that all k trials succeed."

    OpenAI describes the step from one-off checks to repeated ones: "Once you know what "good" looks like, move from individual traces to repeatable datasets and eval runs." Its graders "let you score those traces with structured criteria so you can find regressions and failure modes at scale."

    limits

    Where the platforms run these checks

    OpenAI's trace grading steps are to open Logs and then Traces in the dashboard, inspect a representative workflow trace, create a grader and run it against the selected traces, and use the results to refine prompts, tool surfaces, routing logic or guardrails.

    Microsoft's Foundry evaluators give binary Pass/Fail scores, and its composite Tool Use Quality evaluator (preview) batches five tool checks into one LLM judge call while keeping a score and a reason for each. AWS says its Bedrock AgentCore Evaluations service measures how an agent performs tasks, handles edge cases and stays consistent across inputs. Each vendor's page covers only its own platform.

    Straight answers

    Frequently asked questions about What is AI agent evaluation? How teams test an agent before release

    What is AI agent evaluation?

    It is testing an agent on its intended work and scoring the result and the steps taken. Microsoft says to evaluate "not just the final output, but also the quality and efficiency of each step in the workflow." Source: S3, read 17 November 2026.

    What is the difference between an eval, a task and a trial?

    Anthropic defines an eval as "a test for an AI system", a task as "a single test with defined inputs and success criteria", and a trial as each attempt at a task. Source: S1, read 17 November 2026.

    What is a trace in agent evaluation?

    OpenAI says "A trace captures the end-to-end record of model calls, tool calls, guardrails, and handoffs for one run." Graders then score those traces. Source: S2, read 17 November 2026.

    Is evaluating the final answer enough for an agent?

    Google separates final response evaluation from trajectory evaluation, which looks at "the path (sequence of tool calls) the agent took to reach the final response." Microsoft likewise separates system and process evaluation. Anthropic adds that it is often better to grade what the agent produced than the exact path. Sources: S4, S3, S1, read 17 November 2026.

    How does Microsoft treat an agent run?

    Microsoft's page treats an agent run as reasoning, tool calls and results, and says to evaluate each step as well as the final output. Source: S3, read 17 November 2026.

    How many tasks does a first agent eval need?

    Anthropic says "20-50 simple tasks drawn from real failures is a great start." Source: S1, read 17 November 2026.

    What is the difference between pass@k and pass^k?

    Anthropic says pass@k "measures the likelihood that an agent gets at least one correct solution in k attempts" and pass^k "measures the probability that all k trials succeed." Source: S1, read 17 November 2026.

    Talk to the team that would do the work

    Bring your requirements to a working session with the person who'll actually deliver.

    Book a Discovery Call