How to evaluate AI agents: test sets, graders, regressions

Behind every great AI product is a strong AI agent evaluation process, known as evals, that catches errors early and ensures the agent’s proper functioning. Evals are critical for AI agents executing high-risk actions like refunds on someone’s behalf.

Building that process requires some technical knowledge and data analysis skills. But it’s not nearly as scary or unapproachable as it might sound. To help you get started, we’ve laid out six practical steps for evaluating AI agents and common misconceptions that break the process.

What does it mean to evaluate an AI agent?

Evaluating an AI agent means assessing its performance in completing tasks correctly and whether its decisions meet the organization’s requirements. In practical terms, AI agent evaluations, or evals, are specific metrics that help organizations determine whether the agent works reliably enough for its intended role.

Think about an AI purchasing agent at a manufacturing plant that checks spare-parts inventory and prepares replenishment requests. If an approved part is unavailable or supplier information is outdated, will the agent flag the issue or recommend an unapproved substitute?

And if you change its system prompt to fix that problem, how do you know you haven’t introduced another one?

Without evaluation in place, product managers are essentially left guessing. Thus, evals provide a feedback signal to measure performance, spot recurring problems, and test changes with greater confidence.

That being said, an agent evaluation can examine several aspects of a run:

• The quality of the final result
• The information the agent used
• The tools it called
• Whether it followed the rules governing its behavior

Common AI agent evaluation metrics

Evals work on qualitative and quantitative data. Qualitative evaluation helps you understand what's happening and why, including manually reviewing agent decisions. Quantitative evaluation turns those observations into measurable signals, like counting the frequency of specific failure modes or calculating task success rates.

Here’s the list of fundamental AI agent evaluation metrics:

• Task success measures whether the agent achieved the intended outcome; usually measured in the percentage of runs that produce valid results.
• Action correctness measures whether the agent used tools appropriately, selected valid arguments, and respected permissions.
• Answer quality assesses whether the final output is accurate, complete, and useful. It can include factual correctness, adherence to instructions, and whether the agent supports its decisions with the available information.
• Reliability measures consistency across repeated trials, varying inputs, and edge cases.
• Cost and latency measure the resources required to achieve a successful result, including execution time, token consumption, and tool calls.

Depending on what you're measuring, evals can assess an agent's end-to-end performance, its execution trajectory, or individual components such as tool calls and retrieval steps.

Step 1: Decide what a successful agent run looks like

First, explicitly describe what the agent is supposed to accomplish — the product requirements. Sounds obvious, yet it can be surprisingly difficult. There’s even a name for it: the specification gap.

Here’s how to approach it, according to Andrey Khomutov, our senior data scientist:

“Start with two questions: “What would count as doing the job?” and “What would make a seemingly successful run unacceptable?”

For agents, from a technical perspective, that means checking both the result and how agents got there.”

An agent might produce the correct answer by accessing a leaked answer key or take unauthorized actions while completing its task. Khomutov points out that answer leakage and unsafe actions during evaluations are concrete reasons not to judge the final answer alone.

“I expect trajectory inspection to become a standard part of agent eval suites, rather than something we do only when a result looks suspicious,” he adds.

Then involve a domain expert to validate your success criteria. People with hands-on knowledge of the domain can tell whether the conditions we set for an AI agent reflect a successful outcome.

Step 2. Build a test set from real tasks and failures

Examine actual agent behavior to understand what good and bad performance look like. Resist the temptation to start by generating dozens of tests based only on what you think might go wrong.

Hamel Husain, ML engineer at Parlance Lab, and Shreya Shankar, ML systems and applied AI evals researcher, recommend beginning with error analysis, a technique borrowed from machine learning and data science. The process starts with qualitative analysis of agent execution traces — records showing the user's input, tool calls, retrieved information, and agent output.

• Review individual traces and annotate errors (open coding). Have a domain expert inspect agent runs and write short observations about what went wrong. Focus on the first meaningful error.
• Group observations into failure modes (axial coding). Once you have enough notes, use an LLM to suggest categories for similar errors. These might include incompatible part recommendations, inaccurate inventory retrieval, and failed human handoffs.
• Count the occurrences. Assign each observation to a category, then use a spreadsheet or pivot table to calculate how frequently each failure mode appears.
• Prioritize what to address, considering both frequency and severity.
• Turn important failures into test cases. Save the relevant input, starting conditions, expected outcome, and grading criteria so the agent can be tested against the same scenario again.

How many traces should you examine? Roughly 100, but there’s no magic number. Stop early once you reach theoretical saturation: the point where reviewing additional traces stops revealing meaningful new failure patterns.

Haven’t launched the application yet? Synthetic examples can help build an initial test set, though real user data should inform it as soon as it becomes available.

The use of benchmarks in AI agent evaluation

Beyond creating your own test cases, you can use existing collections of standardized evaluation tasks, known as benchmarks, to assess agent capabilities.

But benchmarks aren't infallible. SWE-bench audits, Epoch AI's benchmark reviews, and correct answers initially rejected in TheoPhysEval illustrate how flawed tasks or expected answers can distort evaluation results.

As Khomutov advises, choose up-to-date benchmarks relevant to your domain and investigate surprising passes or failures before treating the scores as reliable evidence of agent performance.

Step 3. Run the agent in a repeatable test environment

A repeatable test environment is a controlled setup where you can run an AI agent against the same test cases under the same starting conditions and compare the results. This includes a folder of test cases, a Python script that runs the agent, or a set of checks that evaluate the results.

You can start by storing representative inputs as JSON files and using a Python script to execute your agent's workflow and check its outputs with assertions. For our manufacturing plant's inventory agent, a repeatable test environment could look like this:

To set up a repeatable test environment, follow these steps:

• Save representative inputs. Store actual events or test cases in the same structure your application receives in production. In our scenario, you’d save a maintenance request containing the machine ID, required spare part, and maintenance date as a JSON file, along with a snapshot of the inventory and approved-parts data.
• Execute the workflow. Run each input through the agent, including its retrieval steps, tool calls, and response generation.
• Record the result and trace. Capture the final output and relevant intermediate actions (such as inventory lookups) so you can see not only whether the agent failed, but where.
• Repeat cases based on failure severity, frequency, and difficulty of fixing the problem. LLM applications are nondeterministic (they can produce different results from identical inputs), so a successful run doesn't guarantee the next one will succeed. Report the proportion of successful trials and inspect the failures.

For the latter, Khomutov advises measuring pass@1 (how often the agent succeeds on a single attempt) and pass@k (the probability of succeeding at least once across a specified number of attempts). If reporting pass@k, specify the number of attempts, since producing one correct answer among several doesn't mean the agent knows which one to choose.

As your application grows, use a testing framework like pytest to organize and run tests, and consider storing large test datasets and results in a database.

Step 4. Choose the right grader for each check

A grader (also frequently called an evaluator) is a mechanism in the form of a piece of code, an LLM guided by a grading rubric, or a human reviewer that checks an agent’s output or action against a defined criterion. A grader returns a pass/fail judgment and an explanation identifying the relevant evidence.

Info: A grading rubric is a set of criteria and instructions defining what counts as a correct or acceptable result.

The right grader type depends on what you're trying to verify. Here's how the three approaches compare:

Grader

Best suited for

Example

Main limitation

Code-based

Exact, verifiable conditions

Was the recommended spare part on the approved compatibility list?

Cannot reliably assess every subjective decision

LLM-as-a-judge

Context-dependent assessments

Did the agent explain why it couldn't recommend a replacement?

Its judgments can be wrong or inconsistent

Human

Domain-specific expertise and uncertain cases

Does the maintenance manager consider the agent’s recommendation appropriate?

Expensive to apply at scale

Sometimes different aspects of the same run may require different evaluation methods, so you might need to combine graders.

How do you combine graders?

You ask: “Can the result be verified against a known rule or expected value, or does it require human-like judgment?”

• If you can verify the result exactly, start with a code-based grader. Code-based graders can also verify structured outputs, valid field values, tool arguments, or the final state of a database.
• If the check requires interpreting context, use LLM-as-a-judge.
• Use human reviewers for domain-specific or disputed decisions.

Important! An LLM grader can misinterpret evidence or apply a rubric incorrectly. It then needs an additional evaluation step: calibration against human judgments.

Calibrate the LLM grader against human judgments.

Essentially, calibration requires a domain expert to assess actual agent runs. The expert then compares their judgments with the LLM’s and refines the grading rubric based on disagreements.

Here’s a five-step grader calibration process:

• Collect examples of agent inputs, outputs, and relevant traces.
• Have a domain expert independently assess whether each example meets the evaluation criterion.
• Run the LLM grader against those examples.
• Compare the human and model judgments.
• Investigate disagreements and refine the grader's rubric before testing it on new examples.

Khomutov also stresses the importance of testing the LLM grader's consistency separately:

“Give it the exact same saved run and files several times, without rerunning the agent. That lets you separate variation in the agent’s work from variation in the grading.”

If the same output gets different verdicts, don’t assume every score change comes from the agent. Instead, examine a confusion matrix showing when the grader correctly identified errors, missed them, or flagged acceptable outputs incorrectly.

Human review remains essential for establishing reference judgments, resolving disputed cases, and discovering new failure modes that existing graders weren’t designed to detect.

Step 5. Compare versions and run regression checks

Suppose error analysis reveals that your inventory agent frequently recommends unapproved replacement parts. You modify its system prompt to require an explicit compatibility check. The original failing case now passes. But how do you know the change hasn’t affected other tasks?

That’s where regression testing comes in.

Regression testing means rerunning previously successful test cases after changing an agent's prompt, model, or workflow to ensure the change hasn’t introduced new failures.

• Reproduce the original failure using a saved input from the test set.
• Make a targeted change to the prompt, model, or workflow.
• Rerun the failing test to check whether the problem is resolved.
• Run the other tests to verify that the change hasn’t broken previously successful behavior.
• Inspect the results and investigate any new failures.

Compare the baseline and updated agent using the same test cases and grading criteria for accurate results.

Step 6: Monitor AI agent performance in production

Agent evaluation doesn’t stop once you’ve deployed your application. As Husain and Shankar explain, the same evaluators you use during development can also run against real production data to measure how your agent performs over time.

This is known as online evaluation, and it’s one part of a broader production monitoring process that helps uncover recurring and previously unknown failures.

The process is iterative and involves four complementary activities:

• Run calibrated LLM judges on production traces. Sample real agent interactions and apply the evaluators you’ve already validated against human judgments.
• Track failure rates over time. Use the failure categories established during error analysis to quantify recurring problems.
• Keep reviewing traces manually. Automated graders only detect the problems they’re designed to identify. Inspect samples manually of real agent interactions to uncover new errors or unusual user requests.
• Use A/B testing to measure business outcomes (only when appropriate). Compare different prompts, models, or workflows using metrics such as task completion, user satisfaction, and time to resolution. The process helps determine whether a change improves the actual product experience, beyond passing individual tests.

One important note: evals don’t have to be polished to perfection. They simply need to be good enough to improve your product.

Common misconceptions in AI agents evaluation

• Evals are just unit tests. They’re not; unit tests are small, automated checks that are only one part of evaluation. Evals also involve qualitative error analysis, human judgments, production monitoring, and experiments to measure application quality.
• An evaluation platform can do everything for you. Off-the-shelf tools can automate some checks, but they can’t determine which failure modes (types of errors or unwanted agent behaviors) matter to your product.
• You can let an LLM evaluate everything. That’s a great fallacy. Human-led error analysis and grader calibration remain essential, as LLMs lack the business context needed to discover every problem.
• A/B testing or simply using the agent replaces evals. Incorrect. A/B tests can show which version performs better against selected metrics, but cannot, on their own, explain the underlying problems.

AI agent evaluation FAQs

1. What's the difference between AI agent evaluation and LLM evaluation?

LLM evaluation measures how well a language model performs a given task, such as answering a question, summarizing a document, or extracting information. AI agent evaluation assesses a system that independently executes multiple steps to complete a task.

Unlike a typical LLM evaluation, agent evaluation examines both the final result and the sequence of actions taken to achieve it, including tool calls, decisions, and compliance with predefined rules.

2. Key metrics for AI agent evaluation

AI agent evaluation metrics fall into two broad categories:

• Technical performance metrics: Task success rate, action correctness, answer quality, reliability (pass@1 and pass@k), latency, token usage, and cost per successful task.
• Business and user experience metrics: Customer satisfaction (CSAT), issue resolution rate, escalation rate, conversion rate, task completion time, and other outcomes specific to the agent's role.

For example, a customer support agent might be evaluated on successful issue resolution and customer satisfaction, while an AI purchasing agent could be measured by replenishment request accuracy, approval rates, and procurement cycle time.

The right metrics should reflect both how reliably the agent performs and whether its work improves business outcomes.

3. What are the best tools for evaluating AI agents?

Common options include:

• Testing frameworks: Python assertions and are useful for running repeatable test cases and checking exact conditions, such as whether an agent selected the correct tool or returned a valid part ID.
• AI agent evaluation frameworks: DeepEval and Google's agent-eval toolkit support more complex evaluations, including execution traces, LLM-based graders, and automated regression testing.
• Evaluation and observability platforms: LangSmith, Arize Phoenix, and MLflow help teams inspect agent behavior, manage evaluation datasets, compare agent versions, and monitor performance in production.

4. How do you automate AI agent evaluations?

You can automate AI agent evaluations by building repeatable tests, applying automated graders, and integrating those checks into development and production workflows:

• Create an evaluation dataset: Collect representative user tasks, expected outcomes, known failures, and agent execution traces. Include edge cases such as failed tool calls, missing information, and unauthorized actions.
• Automate test execution and grading: Use frameworks like or to run test cases and apply code-based checks or LLM graders. Validate LLM graders against human judgments before relying on their scores.
• Integrate evaluations into CI/CD: Automatically rerun regression tests when developers update an agent's prompts, models, tools, or workflow. Compare results against a baseline and establish quality thresholds that prevent releases when critical checks fail.
• Apply calibrated graders to sampled production traces, track failure rates and other performance metrics, and investigate new errors through manual review.