Testing & Evaluating Agent Workflows Before You Ship Them

Evaluating an AI agent before shipping it means testing three things separately: whether it completes the task correctly, whether it calls its tools correctly, and whether it behaves safely on inputs you didn't anticipate. You do this with a versioned "golden set" of representative and adversarial tasks, a mix of deterministic checks and LLM-as-judge scoring, full execution traces you can replay, and a regression gate that blocks a prompt or model change from shipping if it quietly makes things worse. None of this is exotic — it's the same discipline as testing any other software with a non-deterministic core, just applied to a system that talks instead of returning a fixed value.

Agent workflows are showing up everywhere right now — support bots that can actually take actions, coding assistants that edit and run code, commerce agents that check inventory and place orders, internal tools that chain several LLM calls and API calls together. The failure mode that keeps biting teams isn't a dramatic outage; it's quiet degradation. A prompt tweak that improves one case and breaks three others. A model upgrade that changes tool-calling behavior in ways nobody notices until a customer hits it. An agent that's 95% reliable in demos and unpredictably wrong in the remaining 5%, which is exactly the 5% that erodes trust fastest. Evaluation is how you catch that before your users do.

Why "it works on my prompt" isn't evaluation

Manually running an agent through a handful of scenarios and eyeballing the output is prototyping, not evaluation. It tells you the happy path works. It tells you nothing about consistency across runs, behavior on inputs you didn't think to try, or whether last week's prompt change regressed something that used to work. Because LLM outputs are non-deterministic and prompts are edited far more casually than code, an agent's behavior can drift significantly between two commits that look nearly identical in a diff.

The mental model worth adopting: an agent is a probabilistic function with a large, effectively unbounded input space, wired to tools that have real side effects. You can't exhaustively test it the way you'd test a pure function, but you can — and should — treat it with the same seriousness you'd apply to any system with side effects and partial reliability: a payments integration, a search ranking model, a fraud filter. Those systems all get held to measured, repeatable evaluation before every release. Agents deserve the same treatment, arguably more, because they act autonomously across multiple steps instead of returning one answer.

Step 1: Build a golden evaluation set

Everything else in this guide depends on having a set of representative tasks with known-good outcomes, checked into version control alongside your agent's code and prompts. Treat it as a first-class artifact, not a scratch file. A useful golden set has four kinds of entries:

• Core-path tasks — the requests you expect most often, worded the way real users actually word things (including typos, slang, and incomplete context), each with an expected outcome or a rubric for judging one.
• Edge cases — ambiguous requests, missing information the agent should ask for, multi-step tasks that require chaining several tool calls correctly.
• Adversarial and safety cases — prompt-injection attempts, requests to ignore instructions, requests for actions outside the agent's scope (refunding an order it isn't authorized to refund, deleting data it shouldn't touch).
• Regression cases — every real bug a user or tester ever found becomes a permanent entry here. This is the single highest-leverage habit in agent evaluation: bugs that get fixed but never added to the eval set come back, quietly, the next time someone tweaks the prompt.

Start smaller than feels sufficient — thirty well-chosen tasks that actually get reviewed beat three hundred that nobody looks at. Grow the set continuously as real usage surfaces new patterns, and revisit it whenever the agent's scope changes.

Step 2: Separate what you're actually scoring

A common mistake is collapsing everything into one fuzzy "was the response good" score. Break it into dimensions that can each be tested with the cheapest adequate method:

• Tool-call correctness — did the agent call the right tool, with the right arguments, in the right order? This is deterministic and should be checked with plain assertions against the trace, not an LLM judge. If your agent has function-calling or MCP tool access, log every call and compare it against an expected call graph.
• Task completion — did the end state match what the user actually needed? For structured outputs (an order status, a generated file, a database row), assert directly. For open-ended answers, this is where an LLM-as-judge rubric earns its keep.
• Groundedness / factual accuracy — for anything backed by retrieval or a knowledge base, check that claims in the output are actually supported by the retrieved context, not invented. This catches hallucination even when the final answer "sounds" right.
• Tone, format, and policy adherence — does it stay in brand voice, avoid disallowed topics, follow formatting rules? Usually the cheapest dimension to check with either rules-based checks (regex, schema validation) or a lightweight classifier.
• Safety and refusal behavior — does it correctly decline out-of-scope or harmful requests without becoming so cautious it refuses legitimate ones? Track false refusals as a metric, not just successful blocks — an agent that's too locked-down is a different but equally real failure.
• Latency and cost per task — a technically correct agent that takes twelve seconds and three model calls to answer a one-line question has a real quality problem, just not the kind a correctness score will catch.

Scoring each dimension separately does two things: it makes failures diagnosable (you'll know a regression was in tool-calling, not tone), and it stops one strong dimension from masking a weak one in an averaged score.

Using LLM-as-judge well (and where it lies to you)

For open-ended dimensions like task completion and helpfulness, using a second LLM call to grade the first is the practical default — it's what makes evaluating thousands of natural-language outputs tractable. But an LLM judge is itself a probabilistic component with known biases, and treating its score as ground truth is a mistake teams make constantly.

• Write a specific rubric, not a vague prompt. "Rate this response 1-5" produces noisy, inconsistent scores. "Does the response include the order number? Does it correctly state the refund policy? Does it avoid making a promise the agent can't keep?" produces something you can actually act on.
• Watch for length and position bias. Judges systematically favor longer answers and, in pairwise comparisons, favor whichever response appears first. Randomize order and don't reward verbosity by accident.
• Use a stronger or differently-tuned model as judge than the one being evaluated, where possible, and periodically spot-check judge scores against human review — an eval system nobody audits eventually drifts from reality.
• Keep judge prompts under version control too. A rubric change is exactly as capable of moving your metrics as an agent-prompt change, and needs the same regression discipline.

Trace everything — you can't fix what you can't replay

Every agent run should produce a structured trace: the input, every model call with its prompt and response, every tool invocation with its arguments and return value, and the final output. When something fails in production, you need to be able to pull that exact trace and replay the reasoning, not just see "response was wrong" in a log line. This is doubly important for multi-step or multi-agent workflows, where the failure is often several steps upstream of where it becomes visible — a tool call with a slightly wrong argument that doesn't surface as an obvious error until three steps later.

Store traces long enough to mine them for new golden-set cases. The best source of edge cases isn't imagination — it's last month's production traffic, especially the runs that took unusually long, used unusually many tool calls, or got flagged by users.

Gate releases with regression comparisons, not one-off scores

The single highest-value practice here is running the full golden set against both the current production configuration and the candidate change — new prompt, new model version, new tool schema — and diffing the results, before anything ships. A score of "87% pass" in isolation tells you almost nothing; "87% pass, down from 92%, with three new tool-call failures on the refund-workflow cases" tells you exactly what to look at before you merge. Wire this into CI the same way you'd wire in any other test suite: a pull request that touches a system prompt, a tool schema, or bumps a model version should trigger an eval run and surface the diff in the review, not after a customer reports something odd.

Model provider upgrades deserve the exact same gate as a prompt change. It's tempting to assume a newer model is a strict upgrade; in practice it can shift tool-calling formats, verbosity, or refusal thresholds in ways that break an agent that was tuned against the previous version's quirks. Re-run the golden set on any model swap before flipping production traffic.

Staged rollout: shadow mode, canaries, and human review

Offline evaluation catches what you thought to test for. Real usage always surfaces more. Before an agent takes consequential real-world actions — refunds, order changes, anything touching money or customer data — run it in shadow mode: let it generate the action it would take, log it, but require a human or a simpler deterministic system to actually execute it. This gives you a real-traffic evaluation set with zero blast radius. Once shadow-mode agreement rates are high, move to a small canary of live traffic with tight monitoring before a full rollout, and keep a fast rollback path — a flag to revert to the previous prompt or model version — ready at every stage, not built after the first incident.

For agents built around tool use — including anything wired up through MCP servers — add a specific test tier for tool integration itself: does the agent correctly handle a tool timing out, returning an error, or returning an empty result? Most golden-set failures in production trace back not to the model reasoning badly, but to an agent that was never tested against its tools behaving imperfectly, which is the normal case in any real system.

This kind of AI-native tooling and evaluation discipline is where we're investing next as a company, alongside the design and Shopify work Polo Themes already does — if you're building the visual and interaction layer for AI-assisted or headless commerce today, our Figma UI kits are a solid starting point for the design system underneath it, and we cover related build practices regularly on the Polo Themes blog.

A practical pre-ship checklist

• A versioned golden set exists, includes adversarial and regression cases, and is reviewed by a human, not just the agent that's being tested.
• Scoring is broken into separate dimensions (tool-call correctness, task completion, groundedness, tone/policy, safety, cost/latency) rather than one blended number.
• Deterministic assertions are used wherever the output is structured; LLM-as-judge is reserved for genuinely open-ended dimensions, with a specific rubric.
• Every run produces a replayable trace, and production traces are mined periodically for new eval cases.
• A regression diff between production and candidate runs on the full golden set is required before shipping any prompt, tool-schema, or model change.
• High-stakes actions ship through shadow mode and a canary before full rollout, with a fast rollback path already wired up.

Frequently Asked Questions

How big does a golden evaluation set need to be to be useful?

Smaller than most teams assume, as long as it's genuinely representative and someone actually reviews the results. Thirty to fifty well-chosen tasks covering core paths, edge cases, and known past bugs will catch most regressions. Size matters less than making sure every real bug becomes a permanent regression case.

Can LLM-as-judge fully replace human review?

No — it replaces the *volume* of human review, not the need for it entirely. Use an LLM judge to score at scale, but periodically audit a sample of its scores against human judgment, and keep humans in the loop for anything adversarial, safety-related, or high-stakes.

What's the single highest-leverage practice if we can only do one thing?

Turn every real bug into a permanent golden-set entry and run a regression diff before every prompt or model change ships. This one habit catches the most common failure mode — quiet re-introduction of a previously fixed problem — with the least infrastructure.

Do I need special tooling to trace agent runs, or can I build it myself?

You can start with structured logging — a JSON record per run capturing inputs, every model and tool call, and the final output — well before adopting a dedicated observability platform. The important part is capturing enough detail to replay a failure, not which specific tool you use to store it.