DEV Community

Cover image for AI Agent Testing Cost: What Evals Add to Your Build Budget
Pykero
Pykero

Posted on Originally published at pykero.com

AI Agent Testing Cost: What Evals Add to Your Build Budget

AI agent testing cost usually lands between a tenth and a fifth of the total build, and that money buys three things: a labelled set of real inputs with known good outputs, a way to score the agent against them automatically, and a habit of re-running that score before every change ships. A quote with no line for this has moved the testing onto your customers, who will run it for you in production, one complaint at a time.

What AI agent testing cost actually covers

Testing a normal web feature means checking that a given input produces a given output, and the answer is yes or no. An agent built on a language model gives a slightly different answer every time, calls tools you did not anticipate, and degrades quietly when the model provider updates something. So the testing work is shaped differently, and each of the three parts costs something distinct.

The golden set

The golden set is a collection of real or realistic inputs paired with what a correct response looks like. For a support agent, that is a few hundred actual tickets with the answer a senior agent would give. For a document assistant, it is questions paired with the passage that contains the answer. For an outreach writer, it is prospect pages paired with the facts a human would have pulled from them.

Building this set is the most expensive and most valuable part of the work. It needs people who know the domain, which usually means your team rather than the agency. A good agency will structure the exercise, build the labelling interface, and push you to include the ugly cases: the ambiguous ticket, the page with no pricing, the customer who asks two things at once. Expect this to consume real hours from a product owner or a senior operator on your side. If a proposal assumes the agency can label a few hundred of your tickets alone, treat that as a sign they have not done it before.

Scoring

Once you have the golden set, something has to decide whether each agent output is acceptable. There are three ways to do it, and most suites use all three:

  • Hard assertions. The agent called the refund tool with the right order ID. The output parses as valid JSON. The reply stayed under the length limit. These checks are cheap, deterministic, and catch the embarrassing failures.
  • Model-graded checks. A second model compares the output to the reference answer and scores it on a rubric. This is how you check tone, completeness, and faithfulness at scale. It costs tokens on every run and needs its own calibration against human judgement, or the grader drifts as quietly as the agent does.
  • Human review of a sample. Someone reads a slice of the outputs each cycle. Nothing else catches the failure mode nobody thought to write an assertion for.

Open-source tooling such as promptfoo covers the mechanics of running cases and scoring them. The expensive work happens before the tool is opened: agreeing, case by case, what a good answer looks like for your business.

Regression on every change

The third component has no file you can point to, which is why it goes missing from quotes. Any change to the prompt, the model, or the agent's tools gets scored against the golden set before it ships. If the score drops, the change waits. In our own outreach engine, the regression run is the same set of prospect pages and the same rubric we used to settle the architecture argument described below. A prompt tweak that reads better to a human and scores worse on that set does not go out. That rule is what turns an AI feature from a one-off demo into something you can keep improving without fear.

How much of the build budget it should take

Rules of thumb from our own quoting, offered as opinion rather than survey data:

  • A low-stakes internal agent (summarising meeting notes, drafting first replies a human always reviews) can get away with a small golden set and mostly hard assertions. Budget around a tenth of the build.
  • A customer-facing agent with no human in the loop (a WhatsApp sales agent, a self-serve support bot) needs model-graded scoring and a human review sample. Budget closer to a fifth.
  • An agent that touches money, health, or legal outcomes should have its eval suite scoped and priced as a first-class deliverable, with the acceptance threshold written into the contract. We covered how to write that clause in evals in AI vendor contracts.

On top of the build, there is a recurring cost: tokens for each regression run, and a few hours a month of someone reading the human review sample and adding new failure cases to the set. For most agents this is small compared to the running cost of the agent itself, which we broke down in what an AI agent costs per task.

Evals pay for themselves during the build, not just after

The common framing is that evals are insurance against regressions. They are, but the first return usually arrives before launch, when the suite settles architecture arguments with numbers instead of opinions.

We saw this in our own outreach engine. It scrapes each prospect's website with a self-hosted crawler, runs a local model over the result, and writes one tailored email per company. The natural design was a chain: one step to extract facts about the company, another to pick an angle, a third to draft. We also built the simpler version, where a single call reads the page and writes the email directly, and we scored both against the same set of prospect pages with the same rubric. The single call won on quality and on cost. Without the eval set, the chain would have shipped, because it looked more thorough and everyone on the team preferred it in principle. We wrote more about that comparison in single call vs agent chains.

That is the pattern to expect. The eval suite turns "I think the multi-step version is better" into a table, and the table is often surprising.

What skipping it actually costs

Skipping evals rarely produces a launch-day outage. The damage arrives over months and shows up as headcount.

The agent ships after a week of manual poking and looks fine. Three weeks later someone tweaks the system prompt to fix a complaint, and a different behaviour quietly breaks. Nobody notices for a month because nobody is measuring. The model provider releases an update and response quality shifts. Your team starts to distrust the agent, adds a human review step that was supposed to be temporary, and the economics that justified the project disappear. Eventually someone proposes a rebuild.

Our outreach engine would have followed a version of this path. The three-step chain would have shipped on the strength of a few hand-read examples, cost more per email than the single call, and nobody would have known the simpler design was better until someone happened to try it. Every one of those steps is a cost that never appears on the original quote, which is why the version without evals looks cheaper. The move from pilot to production, which we described in getting an AI agent from pilot to production, is mostly the work of making change safe, and evals are the bulk of that work.

What to ask for in a quote

When you are comparing proposals for an AI agent, these questions separate agencies who have run agents in production from agencies who have built demos:

  1. How large will the golden set be, and who labels it? A real answer names a number of cases and a labelling process involving your team.
  2. What is the acceptance threshold, and how is it measured? You want a score, a rubric, and a statement of what happens if the threshold is not met.
  3. What gets re-tested when the prompt or model changes? The answer should be "everything, automatically", and you should ask to see the script that does it.
  4. Where do new failure cases go? Production mistakes should feed back into the golden set. If there is no path for that, the suite will rot.
  5. What does the recurring cost look like? Token spend per regression run and hours per month for human review.

An agency that answers these fluently has probably been burned by an untested agent before, which is exactly the experience you are paying for. If the answers are vague, read our guide on how to vet an AI agency demo before signing anything.

If you are scoping an agent and want the eval work priced honestly alongside the build, let's talk.


Originally published on the Pykero blog.

Top comments (0)