My Agent Said "All Tests Pass." CI Disagreed Three Minutes Later. Both Were Honest.
The agent says all tests passed. Three minutes later, CI says otherwise. And the part that takes a while to accept: neither of them lied.
The agent ran what it had. The suite it ran was green. CI ran more, in a different environment, against the real integration, and that's where it fell over. Both statements were true. One of them was about a different world than the one being shipped to.
I've decided this failure mode deserves a name, because it's the most dangerous state an agent can be in. Not "failed," that's loud and it pings you. Confidently incomplete. Done-shaped, green, summary says fine, missing the part nobody told it to check.
Why agents misreport done
The mechanics are worth understanding, because they're structural, not fixable by a firmer prompt.
An agent's sense of completion comes from what's in its context. If the tests it can see pass and its task steps are checked off, it's done, honestly and by its own lights. Nobody is deceiving anybody. But the definition of done it's using was implicitly whatever was reachable, and the real definition, the one CI enforces, is everything that has to be true before this merges.
That gap is where whole categories of agent failures live. The unit test suite ran locally. The integration suite didn't exist in the agent's world. The environment differs. The check that silently does nothing still reports success, a failure shape documented in the wild: 1,558 tests ran green on a system where the auth was never actually exercised at all. Green is a claim by the suite about itself. It is not a fact about the world.
So the agent wasn't wrong. It was parochial. It reported completion according to the only definition anyone gave it: the one inside its own context window.
The working fix: make done a contract
Stop letting "done" be emergent. Make it a machine-checkable definition, the same one CI already enforces. Concretely:
The agent's completion step runs the real checks, not its own. Not "the tests I have," the commands CI runs. If it can't run them, it reports "blocked, verification unavailable," which is a better outcome than "done." An honest blocked is information. A false done is a landmine.
Completion reports show evidence, not claims. "Tests pass" is a claim. pytest -q output with the count, the command, the exit code, is evidence. I make the agent paste the artifact into its summary. Verifiable beats plausible, every time.
Done and merged are different states, on purpose. The agent can reach "done by its own definition." Only CI reaches "done by the org's definition." Keeping those labels distinct, and never letting the agent's status overwrite the pipeline's, is most of the fix.
The uncomfortable corollary: this contract is a document, and somebody owns it. If the definition of done lives in three people's heads and a CI config nobody reads, then the agents are all confidently incomplete, all the time, and the only defense so far has been that CI runs last.
What I can't claim
A machine-checkable definition of done only covers what you can check. Taste, intent, whether the feature should exist at all, none of that is contractable, and I'm not going to pretend a checklist replaces judgment. This raises the floor. The ceiling stays human, exactly as it always was.
And there's a failure mode in the fix itself worth naming: contracts rot too. CI changes, the contract lags, and now you have two definitions of done again, one of them wrong and automated. It needs maintenance like any other artifact, which is the price of every real fix in this space.
Stop reading "done" from agents. Read the evidence they attach. If there's no evidence, there's no done, no matter how confident the summary feels.
What does "done" mean for your agent today, and who's checking? Not who's supposed to be checking. Who actually is?
Top comments (5)
A useful completion receipt could bind each check to the exact commit and environment, and distinguish passed, failed and not run with a reason. That would make a different CI image or a newer commit visible instead of treating either green result as universal. How would you detect drift between the receipt and the CI job definition?
The commit/env binding is right, and it's the cheap half: a receipt that says
sha=…, ci-job=test-suite@v14makes a green result non-transferable, so "different image or newer commit" stops being invisible. On drift detection — honest answer: I'd treat the CI job definition itself as the thing under test. Concretely: a canary job that asserts a known-failing condition and expects red, and a known-passing one that expects green, run against the live job definition on every config change. If the canary goes green, the receipt is lying. It's the same move as the field-test hardening pass I wrote about on the Tempo backend: the question for every check is "could this fail if the claim were false?" — asked about the pipeline itself, not just the code.That makes the pipeline itself a testable claim. One extra boundary: a skipped canary or one run against a different commit should be neither green nor red evidence for this change. I’d bind both expected outcomes and the job definition to the same SHA, and make a missing or skipped canary visible as unverified. How would you handle a canary that can’t run in a particular environment?
the "confidently incomplete" name is good. making done mean what CI enforces instead of what the agent can see is the part that stuck with me. I'm building Coppice (getcoppice.app), a desktop app for a team of coding agents, and the human always reviews and accepts the diff before anything lands, for pretty much this reason.
"Done means what CI enforces, not what the agent can see" is the whole article in one line. The human-accepts-the-diff step in Coppice is the right shape too — it keeps the agent's
doneand the org'sdoneas deliberately different states, which is most of the fix. What I'd add from the writing: make the agent paste the actual command, exit code, and output into its summary. "Tests pass" is a claim;pytest -qoutput with counts is evidence. Cuts the confidently-incomplete window a lot.