October 11, 2026
How to Build Reliable Eval Suites for AI Agent Testing in 2026: Coverage, Failure Detection, and Non-Deterministic Assertions
AI agents are now shipping in production — customer support bots that escalate tickets, code assistants that propose refactors, data pipelines that choose their own API calls. Each of these systems has a failure surface that standard unit and integration tests cannot cover: decisions made at inference time, tool calls issued mid-flow, branching logic triggered by model output. Building an eval suite that reliably catches regressions in this environment requires rethinking what coverage means, what constitutes a meaningful assertion, and how to handle the non-determinism that makes AI agent behavior fundamentally different from deterministic software. This post covers the structural decisions that determine whether an eval suite becomes a reliable regression gate or a collection of flaky tests that teams disable.
AI agent failures fall into four categories, each requiring different test coverage. The first is output failure: the agent returns an incorrect, incomplete, or harmful final result. The second is routing failure: the agent takes the wrong decision branch — escalates when it should resolve, calls the wrong tool, or skips a required step. The third is tool call failure: the agent issues a tool call with incorrect parameters, calls the wrong API, or fails to handle a tool error gracefully. The fourth is context failure: the agent loses track of prior conversation state, misinterprets instructions, or fails to carry a variable forward correctly through a multi-turn flow.
Traditional test coverage metrics apply to code paths. AI agent coverage needs to extend to scenario branches: which user intents has the agent been evaluated against, which tool combinations have been tested, and which error states have been exercised. A suite that achieves 100% code coverage on the orchestration layer but tests only the happy path of each intent has low effective coverage of the agent's actual behavior surface. Astaqc's test automation services include AI agent eval design for teams building production agent systems. The complete software testing guide covers how AI agent testing fits into a broader quality strategy.
A complete AI agent eval suite operates at three layers: unit evals on individual prompt-response pairs, integration evals on multi-step flows, and system evals on end-to-end scenarios against a realistic environment.
Unit evals test the model's response to a single prompt under controlled conditions. They are fast, cheap to run, and useful for catching prompt regressions when system prompts or few-shot examples change. A unit eval asserts on the model's completion given a fixed input — does the response include the required JSON structure, does the classification match the expected label, does the refusal trigger on a prohibited input. Unit evals tolerate low variance: they should pass on repeated runs with the same seed and, when asserting on structured output, should use schema validation rather than string matching.
Integration evals test a sequence of steps: a multi-turn conversation, a tool call followed by a second inference, a decision branch that routes to a different agent. They are more expensive to run than unit evals and tolerate more variance — the model may phrase an intermediate response differently across runs while still routing correctly. Assertions at the integration layer should check decision outcomes (which tool was called, which branch was taken) rather than response text. TestInspector's HTTP request steps and variable interpolation support integration-level evals against live API endpoints without requiring a separate test harness. See the TestInspector product page for how these capabilities work in practice.
System evals test the full agent stack against a realistic environment: real or sandboxed external services, realistic user inputs sampled from production traffic, and full conversation contexts. System evals are expensive, run less frequently (typically nightly or on release candidates), and produce the highest-signal regression data. They tolerate the most variance in intermediate steps but assert on final outcomes: was the ticket resolved, was the correct API called, was the output within the acceptable range. Astaqc's software testing services include system eval design and red-teaming for agent systems in production.
The central challenge of AI agent testing is that the same input does not reliably produce the same output. A deterministic assertion — assert response == expected — fails on semantically equivalent responses that differ in phrasing, produces false positives on acceptable variation, and teaches teams to treat test failures as noise rather than signal. Non-deterministic assertions require a different approach at each layer.
For structured output, use schema validation: assert that the response is valid JSON, that required fields are present, that values fall within expected ranges or enum sets. Schema assertions tolerate phrasing variation while catching real structural regressions. For classification tasks, use label matching: the model must return one of a defined set of labels, and the eval asserts on which label was returned, not on the explanation. For open-ended text, use semantic similarity scoring against a reference response using an embedding model or a judge model, with a configurable similarity threshold above which the response passes.
Judge-model assertions — using a separate LLM call to evaluate whether the agent's response meets criteria — are increasingly common for complex assertions that cannot be expressed as schema validation or label matching. A judge model can assess whether a response is accurate, whether it stays within scope, whether it handles an edge case correctly. Judge assertions have their own failure modes: the judge model's behavior is also non-deterministic, and judge prompts require the same regression testing as any other prompt. The practical guidance is to use judge models for assertions that require semantic understanding, use schema and label assertions wherever possible, and run judge evals on a fixed judge model version to avoid cross-model variance contaminating results.
For tool call assertions, assert on the function name and parameter structure rather than parameter values: the agent must call search_tickets rather than list_tickets, the API call must include the customer_id field, the request must use POST rather than GET. Tool call assertions are deterministic in structure even when the agent's phrasing varies. Astaqc's hire QA team services can supply AI testing specialists for teams building eval infrastructure without in-house expertise. The manual vs. automated testing guide covers when manual exploratory testing of AI agents supplements automated evals.
An eval suite that reports only pass/fail on each test case produces limited diagnostic value. Useful failure detection requires tracking additional signals that distinguish one-off variance from systematic regression.
Pass rate over repeated runs is the first signal. Run each eval case five to ten times and report the pass rate distribution rather than a single pass/fail result. A case that passes 9 out of 10 runs is behaving differently from one that passes 5 out of 10, even if both pass on a given CI run. Track pass rate trends across releases: a case whose pass rate drops from 0.95 to 0.80 between releases is a regression signal even if it did not fail outright on any single run.
Latency and token usage are the second signal. A response that becomes 40% slower or uses 30% more tokens without a measurable quality gain is a regression in efficiency even if it passes quality assertions. Track p50 and p95 latency per eval case, and track prompt token counts separately from completion token counts to distinguish input-side bloat from output verbosity increases.
Tool call frequency and distribution are the third signal. If an agent begins making more tool calls per conversation turn, or shifts its distribution of tool calls without a corresponding change in input, that is a behavioral change worth flagging even if the final output still passes quality assertions. Unexpected tool call patterns often indicate prompt drift, context window management issues, or model version changes affecting decision behavior.
Error rate and error type distribution round out the signal set. Track which error types appear in tool call responses, which tool calls time out, and which agent decisions trigger fallback paths. A shift in error distribution between releases is a regression indicator even if the agent handles each error gracefully and final output quality remains stable. Astaqc's manual testing services cover exploratory failure detection scenarios that automated evals do not surface.
A useful eval suite starts with coverage of the top 20 user intents by frequency, plus the top 10 edge cases by business risk. That produces a 30-case baseline that can be run in minutes and catches the majority of regressions from prompt changes. Scale from there by adding cases for intents that have produced real production failures, for tool call combinations that appear in user sessions, and for adversarial inputs that probe the boundaries of the agent's behavior policy. A suite of 100 to 200 well-chosen cases with clear assertions provides more regression signal than a suite of 1000 cases where many assertions are too loose to catch real regressions.
Run unit evals on every commit that changes a prompt, a tool definition, or a model version. Run integration evals on every pull request. Run system evals nightly and on release candidates. The cost of running evals against a live LLM API means that full system eval suites are rarely run on every commit, but the cases most likely to catch regressions in the current change should always run before merge.
Version your expected outputs alongside your eval cases and tie expected output versions to the model version and system prompt version they were generated against. When a system prompt changes intentionally, regenerate expected outputs for the affected cases using the new prompt and review the diffs before committing. Automated expected-output regeneration without human review is a common source of silent regression: the eval suite passes because the expected output was updated to match the regression.
TestInspector's HTTP request steps support GET, POST, PUT, PATCH, and DELETE with request headers, body payloads, and response assertions on status code and response body fields. An AI agent exposed via a REST API can be evaluated using HTTP steps with variable interpolation for test inputs and JSON path assertions on response fields. For multi-turn conversation evals, chained HTTP steps pass the conversation ID and prior response fields as variables into subsequent requests. See TestInspector for trial access and documentation on HTTP step configuration.
An eval suite tests known scenarios with defined expected behavior: it catches regressions from changes and quantifies quality along measured dimensions. A red team tests unknown failure modes: adversarial inputs designed to elicit harmful, out-of-scope, or policy-violating behavior that the eval suite does not cover. Both are required for production AI agent quality. Eval suites run continuously as regression gates; red teams run periodically as exploratory exercises by practitioners who actively try to break the agent. Astaqc's software testing services include AI agent red-teaming alongside eval suite design for teams that need both.
RAG agents require eval coverage at three additional points beyond the base agent: retrieval quality (did the retrieval step return the relevant context), grounding (does the agent's response cite the retrieved context rather than hallucinating), and context window management (does the agent degrade gracefully when the retrieval results are noisy or exceed the context budget). Retrieval quality is measured by recall and precision against a labeled test corpus. Grounding is asserted by extracting citations from the response and verifying they appear in the retrieved context. Context window management is tested by injecting retrieval results of controlled quality levels and asserting on response quality and fallback behavior.
An AI agent eval suite that only checks final output misses 80% of the failure surface. Intermediate steps, tool calls, and decision branches all need coverage — and each requires assertions that tolerate non-determinism without masking real regressions.
October 11, 2026
Sign up to receive and connect to our newsletter
We founded ASTAQC as a tight-knit group of professional Software Testers and software developers.
Subscribe to our Newsletter
