DEV Community

Matt Busel
Matt Busel

Posted on

My coding agent said "done" 1,521 times. Half of those had nothing behind them.

I run Claude Code most of the day, often several agents at once. For about two months I logged every session on my machine and asked one boring question of each "done" message: did a passing check actually run before the agent said it?

A "done" message here means the agent claimed tests pass, a bug is fixed, a build is green, or something is deployed. A passing check means a test, build or lint command ran in that same turn and exited 0.

752 of 1,521 done-claims had no passing check behind them. That's 49%.

What the unbacked ones look like

Most of them aren't lies exactly. They fall into three buckets:

  1. Stale green. The agent ran the tests, then kept editing, then reported success off the old run. This is the biggest bucket by far.
  2. Fixed, never rebuilt. It changed the code the compiler complained about and called the build fixed without building again.
  3. Green with red above it. A small number said "all tests pass" with a failing exit code a few lines up in the same turn. Those are the ones that actually cost me time.

What helped

The thing that moved the number most was boring: a stop hook that refuses to let a turn end on a success claim unless a passing check ran in that turn. The agent then just runs the tests, which is what I wanted in the first place.

It's annoying on docs-only turns, and deciding what counts as a "check" in a monorepo takes some tuning, but it turned "done" back into something I can trust without rereading every diff.

The tool

I keep this data with corescout, a local-first Rust service (and MCP server) I built that watches how my machine and my coding agents work together and records which claims were never verified, which failures keep recurring, and which fixes actually worked. It's MIT licensed: https://github.com/Mattbusel/corescout

I'm curious how other people handle this. Do you treat CI as the only source of truth, make the agent paste command output, use a second model as a judge, or just read every diff?

Top comments (3)

Collapse
 
skillselion profile image
Skillselion •

A stop hook is the right place for this. One detail on how you decide that a passing check ran in the turn: the Claude Code hooks docs say transcript_path is written asynchronously and may lag the in-memory conversation, so it may not include the current turn's most recent messages when the hook fires. A hook that scans the transcript for an exit 0 can miss a check that ran seconds earlier and block a turn that was verified. For the claim text itself the docs point to last_assistant_message. If your hook reads tool results from PostToolUse events written to a side file instead, it avoids the lag and gives you ordering against the last Edit, which your stale-green bucket needs. Which source does it use today?

Collapse
 
rodrigo-duarte profile image
Rodrigo Lemela Duarte •

For regression fixes, our reviewers ask for executed test output. Our Claude Code agents send their work to a Codex reviewer agent, and a claim without output doesn't count. Early this morning an author agent sent a login-theme fix with new tests and a "they pass". The reviewer asked for the executed counter-proof: the new test run against the old commit (it must fail) and against the new one (it must pass), with commands and output. Our manager agent's answer was "fair, the author sent no logs, only the claim", and the request went back to the author.
So, to your question: a second model as judge, but one that asks for logs. After the reviewer's last approval, each engineer runs the full suite once on its own branch, before the batch merges to staging, and CI runs the checks again after the merge. It all happens in an agent-chat room, so the claim and the evidence sit in the same thread. Does corescout distinguish a reported test result from captured command output?

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

A 'done' the agent writes about itself has no evidence behind it. The check that counts runs outside the agent, on the artifact, and reports a number.