Testing AI agents without using another LLM as an evaluator means replacing probabilistic, expensive judgment with assertions that produce the same result every run. Deterministic testing of AI agents is achievable for most of the properties teams actually care about: did the agent call the right tools, in the right order, with valid parameters; does the output conform to a defined schema; did the agent complete the workflow without hallucinating tool names or inventing parameter values. These properties are verifiable without an LLM, and testing them deterministically is faster, cheaper, and more reliable than LLM evaluation.

The instinct to use an LLM as a judge — asking a model to rate whether an agent's output is correct — comes from a real problem: AI agent outputs are often natural language or semi-structured data that does not lend itself to exact string matching. But LLM judgment introduces its own problems: the same output evaluated twice at different temperatures can produce different scores; the judge model can be led by formatting artifacts unrelated to correctness; and the process is expensive relative to deterministic assertions. Most agent evaluation tasks that seem to require LLM judgment can be decomposed into deterministic sub-assertions.

This guide covers the assertion strategies, output contract patterns, and behavioral probe approaches that allow QA engineers to build reliable AI agent test suites without LLM evaluators. The patterns apply to agents built on any LLM provider and framework — LangChain, LlamaIndex, Semantic Kernel, or direct API calls. For the broader context on testing AI systems, see the guide on AI in software testing.

Why LLM-as-Judge Creates Evaluation Problems

Using one LLM to evaluate another's output is sometimes described as the only option for assessing quality or correctness of natural language outputs. This overclaims. Quality and correctness are composite properties; many of their components are deterministically verifiable, and the remainder — genuine semantic quality of free-form prose — is often not what agent tests actually need to verify.

LLM-as-judge evaluation has four practical problems. First, it is non-deterministic: the same response evaluated twice produces different scores due to temperature settings and model updates, making test results unreliable for regression detection. Second, it is expensive: an evaluation pass that calls a frontier LLM for each test case multiplies API costs by the number of evaluations, often making large test suites economically impractical to run on every CI trigger. Third, it is slow: LLM calls add latency to each evaluation, extending CI pipeline duration. Fourth, it is biased by presentation: LLM judges tend to rate well-formatted, verbose outputs higher regardless of factual accuracy, introducing systematic evaluation error.

The practical alternative is not to avoid evaluating AI agent outputs — it is to decompose what correct means into components that can be verified separately. An agent that retrieves a customer record and formats a summary email can be tested for: did it call the correct retrieval tool; did the tool call include the required parameters; did the output include the customer's name; did the output avoid including fields the customer has marked private. Each of these is a deterministic assertion. The question of whether the summary is well-written may not need a test at all if prose quality is not a business requirement.

When LLM Evaluation Is Justified: The Narrow Remaining Cases

After applying deterministic assertions, output contracts, and tool call verification, the remaining cases that genuinely require LLM evaluation are narrower than they appear. Three categories legitimately benefit from LLM judges.

Factual accuracy evaluation: verifying that a RAG agent's response accurately reflects the retrieved context, without fabricating or contradicting it. This is a semantic claim that exact matching cannot verify. A targeted evaluation call — does this response accurately reflect the provided context, answer yes or no with a one-sentence justification — is appropriate here and cheaper than an open-ended quality rating.

Tone and policy compliance: verifying that an agent's response does not violate content policies, communication guidelines, or regulatory constraints. A deterministic keyword blacklist handles obvious violations; a small LLM evaluation call handles borderline cases requiring semantic understanding.

Semantic consistency: verifying that an agent produces equivalent responses to inputs that mean the same thing but are worded differently. This catches prompt brittleness that identical-input testing misses.

The table below maps assertion types to the appropriate evaluation approach:

Building a Deterministic Agent Test Suite: Practical Setup

A deterministic AI agent test suite requires three components: a way to call the agent under test in isolation; a way to capture tool call logs; and an assertion framework to verify output and behavior.

For Python-based agent frameworks, pytest combined with a JSON Schema validation library (jsonschema) and a custom tool call capture wrapper covers the majority of assertion types. The test harness wraps the agent call, intercepts tool invocations, and returns a result object containing the agent's output, the tool call log, timing data, and any structured output fields. Assertions run against this result object.

For teams using TestInspector, the HTTP request step covers the API layer of agent testing: call the agent's API endpoint, assert on the response status code, verify that required fields are present in the JSON response body, and chain multiple steps to test multi-turn conversation sequences. The test automation services at Astaqc handle agent test harness configuration as part of setup engagements.

For CI integration, agent tests run as part of the standard pipeline but require a model API key in the CI environment. The practical trade-off is test coverage versus cost: running the full agent test suite on every commit is feasible for small suites; larger suites typically run on pull request merges or nightly schedules. See the guide on how to outsource software testing for how to structure QA coverage when internal capacity for agent testing is limited.

Deterministic Assertions for AI Agent Outputs

Deterministic assertions for AI agents follow the same pattern as API testing: define what the response should contain, verify it, pass or fail. The difference is that agent responses are often structured data wrapped in conversational text, so assertions need to extract and verify specific properties rather than matching the full response string.

Schema assertions verify that the output conforms to a defined structure. Agents forced to produce JSON output — through system prompts, response format parameters, or structured output APIs — can be validated against a JSON Schema definition. A schema assertion that checks for required fields, field types, and value constraints catches the most common agent output failures: missing fields, wrong types, and out-of-range values.

Presence assertions check that specific values, substrings, or patterns appear in the output. A customer record summary should contain the customer's name. A code generation agent's output should contain a function definition. A search agent's response should reference the query term. Regex patterns handle cases where exact string matching is too brittle.

Absence assertions check that specific values do not appear. An agent processing a customer record marked confidential should not include the account number in its response. A summarization agent should not reproduce verbatim text from a source document beyond a defined length threshold.

Tool call assertions verify that the agent invoked the expected tools with valid parameters. Most modern agent frameworks — LangChain, LlamaIndex, OpenAI Assistants, Anthropic's tool use API — provide access to tool call logs. An assertion that checks whether the agent called search_customer_records before draft_email, and whether the search call included the customer_id parameter, is a precise behavioral test that does not require evaluating output quality.

State transition assertions verify that the agent reached an expected state. For agents that maintain conversation or workflow state, asserting the state after a defined input sequence is a deterministic way to verify correct behavior without evaluating the prose of each response. The test automation services at Astaqc cover AI agent test harness setup for teams integrating these patterns.

Frequently Asked Questions

Can I test AI agent behavior without access to the agent's internal tool call logs?

If the agent framework does not expose tool call logs, testing shifts to output assertions only: schema validation, field presence, and pattern matching on the response. This is less precise than tool call verification but still deterministic. Most modern frameworks — OpenAI function calling, Anthropic tool use, LangChain, LlamaIndex — expose tool call data in the API response or through instrumentation hooks. If the agent is a third-party service that does not expose internals, black-box output contract testing is the available option.

How do I test a streaming agent response deterministically?

Collect the complete streamed response before asserting against it. Streaming is a delivery mechanism, not an evaluation constraint. Accumulate the token stream into a complete string, then run schema, presence, and absence assertions against the full response. Tool call assertions are typically available before streaming completes in most framework implementations, through tool call blocks or events that arrive before the final text response.

What is the minimum test suite for a production AI agent?

A minimum viable deterministic test suite covers three categories: happy path tool call sequence (correct tools called in correct order for standard inputs), output schema compliance (all required fields present, no forbidden fields), and at least three adversarial inputs designed to trigger off-nominal behavior — ambiguous queries, missing required parameters, inputs that match common failure modes. This suite runs fast, produces stable results, and catches the most common agent behavior regressions.

How do I handle non-deterministic agent outputs in regression testing?

For non-deterministic agents, regression testing shifts from exact output matching to property-based testing: verify that output properties are stable — required fields present, schema valid, tool calls correct — rather than that outputs are identical. For outputs where text content matters, maintain a golden dataset of accepted outputs and run semantic similarity checks only when exact matching fails. This tolerates natural variation in LLM outputs while still catching genuine regressions.

Should agent tests run against production or a staging environment?

Agent tests that depend on production data or production API keys introduce risk if they write state. Read-only agents — information retrieval, summarization, classification — can run against production for realistic coverage. Write-side agents — those that send emails, create records, or trigger external actions — should run against staging with test accounts. The AI in software testing guide covers environment strategy for AI system testing in more detail.

How often should AI agent tests run in CI?

Test cadence depends on cost and stability. Tool call assertions and schema validation tests are cheap and fast — run these on every commit. Larger end-to-end agent test suites with multiple tool call sequences cost more per run — schedule these on pull request merges or nightly. For agents under active development, daily full-suite runs catch model behavior drift that weekly runs miss. The Astaqc automation services team configures CI integration for agent test pipelines as part of setup engagements.