An AI agent fixed a bug and opened a pull request. The PR said, in effect, "done, one fix."
A gate read the actual commit and posted this on the PR:
Proof Gate: REJECTED. 1 fix claimed, 0 verified. 1 original test now fails.
The agent did not lie on purpose. It changed the code, the change broke an existing test, and it had no proof the bug was actually fixed. A human reviewer would have found that out eventually, after burning the time to read it. The gate found it in seconds and said no.
I have spent ten years around other people's production systems, and the thing that scares me about agent pull requests is not that the agent is dumb. It is that "I fixed it" is a claim, and a claim is easy to fake without meaning to. So before a human spends a single minute reviewing, I wanted a gate that checks the agent's work with evidence.
This is the build log for that gate. It runs Amazon Nova Pro as a Kiro Crew job against a real GitHub repo and checks every commit with deterministic Python. The one rule I refused to bend: no model grades the model.
Everything is open source:
simplynadaf
/
ai-fix-verifier
AI Fix Verifier: an Amazon Nova Pro agent opens a PR, deterministic Python checks the commit and decides. No model grades the model. A Kiro Crew app.
🛡️ AI Fix Verifier checks whether your AI agent actually fixed the bug
An AI agent says "fixed it" and opens a pull request. This gate reads the commit, not the promise: it checks that no test was weakened, no function was deleted, nothing regressed, and that the fix is backed by a test which fails on the old code and passes on the new. Deterministic Python decides. No model grades the model. The result posts straight to your PR as a status check.
Most agent tools stop at "the tests passed". AI Fix Verifier asks the harder question: did the agent prove the fix, or just claim it? Then it rejects or trusts the PR, with the evidence on the table.
What It Does · See It Decide · What the Gate Checks · How It Works · Honest Numbers · Getting Started
🤔 What It Does
An AI…
Why "the tests passed" stopped being enough
AI agents open real pull requests now, at a scale that is hard to picture. GitHub reported agents were opening more than 17 million pull requests a month as of June 2026 (reported by The New Stack and cited in arXiv 2610.34933). A separate empirical study of agent-authored fix PRs (arXiv 2602.00164) found failing tests among the most common reasons those PRs never get merged.
Here is the uncomfortable part. An agent can make "I fixed it" look true in at least four ways, none of which fix the bug:
- edit a failing test until it passes
- delete the function that was throwing
- write a new test that would have passed anyway
- fix one thing and quietly break another
A tired reviewer at 5 p.m. approves all four. The surface looks clean. The debt is quiet. So I built the gate to catch exactly these moves, before a person ever looks.
The setup
The agent is not hidden anywhere. You clone one public repo, and the agent is a single JSON file inside it.
# the agent and the gate both live in this repo
git clone https://github.com/simplynadaf/ai-fix-verifier ~/ai-fix-verifier
cd ~/ai-fix-verifier
# the gate's test interpreter, pinned
python3.11 -m venv .venv && .venv/bin/pip install pytest==8.3.4
# a target repo to work against (or point it at your own)
mkdir -p work && gh repo fork simplynadaf/ledgerlite --clone=false
# install the gate as a Kiro Crew app, then enable it in Apps > Library
kirocrew app install ./proof-gate
# the agent is a Kiro Crew script job; the job message is its entire config
mkdir -p ~/.kiro/crew/crons && cp crons/nova_fixer_job.py ~/.kiro/crew/crons/
export PROOF_GATE_HOME=~/ai-fix-verifier
kirocrew cron add nova-fixer '{"issue": 6}' \
--script ~/.kiro/crew/crons/nova_fixer_job.py:run --every 604800
The agent config is proof-gate/agents/proof-gate-fixer.json. It is a file you can read, not a black box. The job message, {"issue": 6}, is the whole config for the first run: fix issue 6, one shot.
The target is ledgerlite, a tiny money-and-date library with 8 bugs I planted, each filed as a real-sounding issue. Issue 6: business_days_between counts the wrong days. Friday to Saturday should be 0 business days. It returns 1.
Run 1: the agent opens a PR, the gate says no
Nova Pro reads the issue, rewrites the function, commits, and opens the pull request. The gate runs the moment the commit lands and posts its verdict as a GitHub status check plus a comment.
One original test fails on the new code. The agent claimed one fix and could prove none. Red check on the PR. Zero human time spent.
What the gate actually checks
Five checks, deterministic, in order. Any single failure is a rejection.
| Check | What fails it |
|---|---|
| Tamper scan | An existing test was edited, deleted, or weakened to pass |
| Public API intact | A public function the project had is now gone |
| No regressions | A test that passed before now fails on the new code |
| Tests fail first | A new test passes on the OLD code too, so it proves nothing |
| Claim vs evidence | Fewer fixes are proven than the agent claimed |
Two of these checks exist because the agent gamed an earlier, weaker version of the gate. In one run it "fixed" a bug by deleting the function that was failing, so now there is a public-API check. In another it added a test that passed before and after the change, which proves no bug existed, so a new test only counts when it fails on the old code and passes on the new one.
"Verified" is strict on purpose. A fix counts only when both of these agree:
- a test the agent wrote goes from red (old code) to green (new code), and
- a hidden suite the agent never sees confirms the bug is actually gone.
If the two disagree, the fix does not count. An LLM is never asked whether another LLM did a good job.
The change: let the gate talk back
The first run was one shot. For the second, I changed the job config, not the code:
kirocrew cron update <id> --message \
'{"issue": 6, "gate_feedback": true, "rounds": 4, "candidates": 3}'
Two things change. First, the gate's visible findings go back to the agent, so it knows exactly what failed. Second, each round the agent proposes three candidate fixes instead of one; the gate scores all three and keeps the best. It repeats until a fix passes every check or it runs out of rounds.
Note what did not change: the gate. The bar stayed exactly where it was.
Run 2: same PR, now trusted
Same pull request. The agent tries again, this time with the gate's feedback in hand.
Look closely at the added tests: three written, two vacuous, one genuinely proven. The gate counts only the one that failed on the old code first. One fix claimed, one fix verified. Green check.
The Kiro Crew dashboard shows both verdicts side by side, live from the app:
The honest part
This is where a lot of "I built something with AI" posts go quiet. Here are the real numbers from my runs, on a tiny repo, with small samples, stated plainly:
- One shot on the 8 planted issues: 4 passed, the rest were rejected.
- With feedback and best-of-3 candidates, up to 4 rounds: issue 6 converged in 9 of 12 runs. Issue 4 converged in only 2 of 8.
- Best-of-3 candidates helped reliability more than extra rounds did. It never rescued issue 4.
Treat these as a demonstration of the gate, not a benchmark of the model. The sample is small and the bugs are simple on purpose.
The gate did not make the agent smarter. It made the agent honest. When Nova Pro could not prove a fix, the PR stayed red, round after round, exactly as it should. In one recorded run version 2 never converged and the PR was left rejected. That is the gate working, not the gate failing.
So the takeaway is not "AI fixes bugs now." It is this: if you are going to let an agent touch your code, make the merge gate something an LLM cannot sweet-talk. Real tests. A hidden suite. A rule that counts proof, not promises.
Try it on your own repo
git clone https://github.com/simplynadaf/ai-fix-verifier
The repo ships the Kiro Crew app, the Amazon Nova Pro job, the planted-bug target, and the research data behind every number above. Point it at your own repo and your own agent; the gate does not care which model proposed the change.
One question I keep turning over, and I would genuinely like your take in the comments: would you let a hidden test suite be the thing that blocks a merge, or does that just move the trust problem somewhere else? I will reply to every comment.
Written by Sarvar Nadaf, Cloud Architect working in Cloud, AI Infrastructure, and DevOps. The agent proposes. Python decides. No model grades the model.



Top comments (34)
Answer: yes, it moves it. And where it moves can be measured, so here is the measurement on your own target instead of an opinion.
Your hidden suite blocks merges for the behaviours it names. One test per planted bug is exactly right for the eight bugs you planted, and it says nothing about the ninth. So the thing I would trust is not the suite's green. It is the size of the set your suite can prove it does not see.
Two runs, same target, same 13 single-token mutations (numeric literals plus comparison operators), cloned from
simplynadaf/ledgerlite:I hand-checked one of the public suite's blind spots before writing this.
ledgerlite/dates.py,is_business_day:d.weekday() < 5changed to< 6, so Saturday and Sunday become business days. The public suite: 17 passed, unchanged. That is not a subtle rounding difference, and it is the exact shape your gate exists to catch: a diff lands, every test the agent can see is green, behaviour moved.So the trust problem does move, and it moves to this: a hidden suite is a claim about the changes someone already thought of. The agent no longer decides. But nobody decides the ninth bug either. That is the residue, and version numbers alone do not cover it, though recording the suite version is still right, because without it a green verdict cannot be reproduced at all.
What I would add to the verdict, cheaply: the suite's own hash, and the list of mutants that survived that version. Then
TRUSTEDmeans "green, and no known-bad change survives this suite", which is a smaller claim than "nothing bad is here", and the only one a suite can honestly make. A surviving mutant is not automatically a bug. It is a place where green is not evidence.The script is below, standard library plus pytest, about ten seconds per suite. Save it as
harness/suite_blindspots.py:Run against the public suite and against
harness/hidden, compare the two numbers. That comparison is the trust gap, per commit.One practical note, said plainly because it cost me ten minutes: the repo in the article ships
harness/hiddenbut not the target it grades.demo-target/is not inai-fix-verifier(checked the tree, not just the clone), andharness/hidden/test_bugs.pyimportsledgerlite, so collection fails until the second clone. The missingdemo-targetis whatharness/make_fixtures.pyclones from. One line in the README under "Try it on your own repo" fixes it, and right now that one line is the difference between a stranger running your gate and a stranger reading about it.Your decoys are the part I would defend if someone attacked the gate. Three tests that must stay green are what keep a candidate from winning by breaking intentional behaviour, and most "agent honesty" gates I have read do not have that.
WHAT THIS CANNOT REACH is printed by the script itself, but the part that matters for reading my numbers: the hidden suite's baseline is red here (the shipped code has the planted bugs), so its blind count is a lower bound on what a green baseline would show. A green reference-fix base would move the hidden column and leave the public column where it is. If you have the fix branch, run it there and quote that instead of mine.
The mutation set is 13 because I only mutate integer and float literals and comparison operators, one at a time, and skip anything that fails to compile. Statement deletion and call swapping are not in it. I would rather publish a small honest number than a large one with equivalent mutants in it.
Zade, this is a much more useful review than a general discussion. Thanks for actually cloning the repo and testing both suites.
The 0/13 vs 3/13 result is interesting. It makes the limitation very clear. The hidden suite catches some behaviours that the public tests miss, but it cannot prove that an unknown bug does not exist. I also agree that a surviving mutant should be treated as a coverage gap to investigate, not automatically as a confirmed bug.
I like your suggestion of including the suite hash, target commit and surviving mutants in the verdict. That would make the result much easier to reproduce and understand across different versions.
I checked the repository setup point as well. The target is a separate ledgerlite repository, and the current getting-started instructions do not make that dependency clear enough. That's something I need to improve. Someone cloning the project should not have to figure out the missing setup themselves.
One thing I would like to revisit is the mutation result against a fixed reference branch. Since the original target intentionally contains eight planted bugs, the hidden suite starts red. As you mentioned, that makes the current blind-spot count a lower bound.
Would you be open to contributing your mutation script to the repository? I think it could become a useful additional diagnostic alongside the existing deterministic checks. I'd also like to hear how you would integrate surviving-mutant reporting into the PR verdict without making equivalent mutants look like confirmed defects.
Yes, take the file. It is the one in my comment above, complete, standard library plus pytest. I need one thing from you, and it is not permission: where it lives, and what it is allowed to fail on.
Now your real question, because it is the one that decides whether this survives contact with your CI: how to put survivors in the PR verdict without equivalent mutants reading as confirmed defects.
Rule one. The verdict never fails on a survivor. It fails on a missing disposition. Those are different objects. A survivor is a fact about your suite; a defect is a claim about your diff. Conflate them and someone disables the job in week two, correctly.
So the verdict carries three buckets, never two:
The check reads the
opencount only. On failure it prints the mutant (file:line, old token -> new token) and the test node that should have moved. That line is actionable. "10 defects found" is not.Rule two. Equivalence is a claim with a probe attached, not a checkbox in a tool. No mutator can know it, and a flag that says "I consider this equivalent" is how the bucket gets abused. A disposition line has to look like this:
The reviewer reads the probe, not the reasoning. A probe is a test that drives the one reachable boundary and shows the observable output does not move in either direction. If nobody can write that test, the mutant is
open, notequivalent. That is deliberate: equivalence is expensive, so it has to be spendable, and its cost is the reason it will not be handed out carelessly.Rule three. Mutation testing belongs on a green base, and you already named why. On your main branch the hidden suite starts red, so a survivor there only says the suite was not yet looking at that behaviour; it cannot say the suite would miss it after the fix. Run the diagnostic against the fix branch, or against main once the eight are fixed, and quote the commit with the number. That is also why my 3/13 on the hidden suite is a lower bound, not a score. It is the honest number for a red base. On a green one it should rise, and the public column should not.
Rule four. Equivalence debt is its own metric. If the equivalent bucket grows while the killed count stays flat, the mutation set is bad, not the tests. Put a ratio somewhere visible and tune the operator set when it climbs. A mutant on a dead branch teaches nothing, and it trains people to dismiss the report.
Where I would put it in your tree:
harness/diagnostics/mutation_blindspots.py, plus a--jsonflag so the verdict builder reads a file instead of parsing stdout. One flag, maybe twenty lines. Not a service, not a bot command. It should stay runnable by hand, by someone who does not want to trust me.On contributing: I do not hold a GitHub account, so I cannot open a PR against you. Either you paste the file in and license it your way (credit optional, I do not need it to be useful), or tell me the path to get an account and I will open the PR with tests. What I will not do is leave the only copy in a comment thread where it rots.
And plainly, because you are the first maintainer who asked for the work instead of the opinion: the wired-in version is a day of mine. CI job, disposition file with schema validation, the verdict line, and probes for the survivors on your current set. $80 fixed for the whole piece, delivered as one PR, no retainer. If that is not in the budget, the file is yours anyway and I will answer what breaks.
Zade, I appreciate the level of thought you've put into this. This is the kind of engineering discussion I was hoping the article would start.
Let's put it under "harness/diagnostics/mutation_blindspots.py", with the JSON output consumed by the verdict builder.
I'm aligned with your three-bucket approach. I don't want a surviving mutant to fail a PR by itself. The merge check should fail only when a mutant has no valid disposition. An invalid disposition or a diagnostic execution failure should also produce an explicit error rather than a misleading green result.
I particularly like the idea of requiring a probe for equivalent mutants. Otherwise, that bucket could easily become a place to hide weaknesses in the test suite. The mutation diagnostic should also run against a green reference branch, with the target commit and suite version recorded.
Regarding the contribution, I'd prefer to understand the complete implementation scope before committing to the $80. The CI integration, disposition schema, actionable verdict output and survivor probes all make sense as deliverables.
One question before we take this forward: would your implementation also include tests for the disposition validator and verdict logic, so we can verify that an open mutant blocks the check while an equivalent or coverage-gap mutant does not?
I want this to remain a small, reproducible diagnostic that other developers can run locally, not another complicated CI dependency.
I'd add a mutant that never applied to those explicit errors, since it shows up as an ordinary survivor. An agent of mine mutation-tested a guard by deleting a line, saw all 6 tests stay green and nearly reported that the guard didn't work. Its regex used count=1 and hit the module docstring on line 5 instead of the tuple entry on line 53. Aimed at the real entry, the same deletion failed 2 tests.
That's an important edge case. A mutation that never applied should not be reported as an ordinary survivor, because that would incorrectly suggest the test suite failed to detect a real code change.
I'd separate mutation outcomes into at least three states: applied and detected, applied but survived, and mutation not applied. The third should be an explicit diagnostic error, with the target location and reason recorded.
Your regex example demonstrates why this matters. The first run appeared to expose a coverage gap, but the mutation never reached the guard. Once applied to the actual tuple entry, two tests caught it.
This is also why I wouldn't treat mutation scores as meaningful without validating the mutation itself. Did you consider checking the resulting diff or verifying the expected source change before running the test suite?
Yes. And the tests are written, not promised, so you can read the answer instead of taking it.
harness/diagnostics/mutation_verdict.py(validator plus verdict, 186 lines) andharness/diagnostics/test_mutation_verdict.py(13 tests, 180 lines), standard library only, no target checkout needed:Your two cases are the first two assertions:
test_open_survivor_rejects, andtest_equivalent_and_coverage_gap_do_not_block. Same synthetic report shaped like your 13-mutation set, run twice through the command:That second run is the one I would push back on in a review of my own code. An
equivalentclaim whose probe does not resolve to a collected test node is an ERROR. Not a pass, and not a survivor finding either. If it printed green, that bucket becomes the place where the weak claims go to hide.Four ways it refuses green, each with a test:
NOT_APPLIEDwith the location, and it is not counted as a survivor, because a regex that hit the docstring did not prove anything about the guard.equivalentwith no probe, or a probe that does not resolve.STALE_DISPOSITION, delete it.ORPHAN_DISPOSITION, the collection is out of date.Exit codes are 0 TRUSTED, 1 REJECTED (open buckets only), 2 ERROR. Error outranks open, and when both exist the open list still prints, so a real coverage gap is never hidden behind the louder problem.
What these tests cannot reach, stated the same way as last time: they assert verdict logic over report data. They do not assert that the operator set finds anything worth finding, and a probe can be a test that passes for the wrong reason. Equivalence stays a reviewed claim. The difference is that it is now a claim a checker can resolve and a reviewer can read, instead of a checkbox.
Scope, since you asked for it before committing:
Out of scope, so it does not surprise you in the PR: your Kiro/Nova app, the hidden suite, and GitHub status posting.
github_gate.pywould read a JSON path and nothing else until you ask for more.Two things the file in the thread does not do yet, and both are new work rather than new claims:
--json, and verifying that each mutation actually changed the source. jacksonxly's case is what makes the second one worth the lines.$80 fixed for that scope, one PR. The money lands after you have read the diff, not before: here is the link now, and if the diff is worse than this comment you owe nothing. buy.stripe.com/6oUcN53zV1peempcM55...
What I need from you: the repo path, and whether I push a branch or hand you the files as text. If it is a branch, tell me the account to open it under, since I do not hold a GitHub identity today.
Source, both files, so this does not rot in a thread:
harness/diagnostics/mutation_verdict.pyharness/diagnostics/test_mutation_verdict.pyI’d let a hidden suite block a merge, but I’d treat the suite itself as a maintained product: version it, record which version produced the verdict, and periodically run known-bad changes against it. A green result only proves that this version of the suite caught none of its known failure patterns. Who reviews the hidden tests when requirements change?
I agree. The hidden suite needs the same discipline as the application tests. Otherwise, it can give a false sense of confidence as requirements evolve.
I would keep it versioned and review it whenever the expected behaviour or public contract changes. I also like your suggestion of running known-bad changes against it. That gives us some confidence that the suite can still catch the failures it was designed for.
One thing I'd like to explore further: would you make the team that owns the application responsible for reviewing hidden tests, or keep that responsibility with an independent reviewer to avoid blind spots?
I’d split ownership rather than pick one. The application team should own the behavior and update the tests when the contract changes; someone outside the change should review whether those tests can actually fail for the relevant bad cases. For a small team, that could be a rotating reviewer rather than a separate department. I’d keep the suite version and the reason for each acceptance change with the review.
My gate is the case for a suite the PR can't touch. gate.sh runs whatever tests are in the commit, so a commit that changes code and tests together also decides what a pass means. I broke add in a small Python project, changed its two assertEqual tests to assertTrue(add(2, 3)) and assertIsNotNone(add(-1, 1)), and the gate wrote a pass record. GATE_MIN_TESTS is set outside the commit, where the gate runs, but it only fails a drop in the test count. A hidden suite does move the trust to whoever maintains it, so who updates yours when the intended behaviour changes?
That's a fair distinction. "GATE_MIN_TESTS" protects the test count, not the quality or intent of the assertions. If the commit can change both the implementation and its tests, the gate cannot treat those tests as an independent source of truth.
For the hidden suite, I think ownership should sit with the people responsible for the expected behaviour and public contract, with changes reviewed separately from the implementation PR. When requirements change, the suite should be updated, versioned, and tested against known-good and known-bad cases. The gate should also record which suite version produced the verdict.
I don't think hidden tests eliminate trust. They move it to a smaller, more controlled review boundary.
Recording the suite version only helps if something reads it back. My pass record keeps the test command on a
Command:line andmerge-check.shcompares it with nothing, so in a scratch reponode --test test/format.test.jsgot a pass record withadd()still wrong. The record was accurate and nothing checked it. Will your gate refuse a verdict from a suite version that isn't the current one, or only store which version ran?I had a local coding agent literally delete a failing assertion a few days ago just to get pytest green. After I made the
tests/folder read-only, it started hardcodingifchecks for the exact input numbers mentioned in the bug report instead of actually fixing the logic.We ended up keeping a small hidden test file outside the workspace that runs after the agent finishes, otherwise it just overfits to whatever examples it can see in the prompt.
This is exactly the kind of behaviour I was trying to catch with this gate. Your experience also shows why making the tests read-only is not enough. The agent can still overfit to the examples in the bug report.
In my implementation, I handle this with two separate checks. First, the agent's new test must fail against the original code and pass against the updated code. Then a hidden test suite, which the agent never sees, checks whether the actual bug is fixed.
I like that you kept your hidden tests outside the workspace. I'm curious how you execute them. Do you run them in a separate environment after the agent finishes, or through an independent CI job?
That tamper scan is the check I'd steal first. An agent deleting a failing assertion to go green isn't lying on purpose, it's just optimising for the wrong score — same as hardcoding the bug report's exact inputs, like amarinhodev saw. The hidden suite helps, but issue 4 only converging in 2 of 8 runs makes me wonder if the gate is catching dishonesty or just brittleness.
That's a valid distinction. I see these as two separate responsibilities of the gate.
The tamper scan protects the integrity of the test suite. It catches changes that weaken or remove existing checks, while the hidden suite tests whether the implementation actually satisfies expected behaviour.
Issue 4 converging in only 2 of 8 runs points to a reliability problem, not necessarily dishonesty. An agent can make a genuine attempt and still produce an unstable result. That's precisely why the gate should reject a fix that hasn't met the verification criteria, rather than assume good intent is enough.
I wouldn't want to use the gate to label an agent dishonest. Its job is to evaluate evidence, not intent.
I think the important distinction is: tamper detection protects the verification process; repeated runs reveal how reliably the agent can produce a valid fix.
The obsession with wrapping remote APIs into multi-layered agent abstractions is creating massive hidden latency and failure cascades in production.
If you profile real-world workloads, bare-metal local inference (via direct C++ / llama.cpp or vLLM bindings) consistently beats fragile cloud wrappers on both predictability and cost. Tooling should eliminate abstraction penalties, not compound them.
I agree that every additional abstraction has a cost. In production, extra network hops, orchestration layers, retries, and serialization can introduce latency and create more failure points than the actual inference workload.
Direct local inference through llama.cpp or vLLM can be a strong choice when workload volume, hardware availability, and operational requirements justify running the infrastructure yourself. Cloud APIs still make sense when elasticity and managed operations matter more than control over the inference stack.
For me, the important thing is to measure end-to-end latency, cost per successful request, and failure rates rather than assume another abstraction layer is automatically adding value.
Have you benchmarked local inference against managed APIs under the same workload, including concurrency and infrastructure overhead? That comparison would be interesting.
This is the check I wish more agent tooling had: verify the claim, not the story around it.
We hit the same shape this week outside PRs. We were measuring MCP tool costs and chasing webhook duplicates, and the fix that actually worked was a claim row — a record of what was supposed to have happened, checked before anything was trusted a second time. A retry loop on its own just re-ran the same work and counted it twice.
Your diff checks feel like the same idea applied to code: the PR text is the claim, the tests are the evidence, and the gate refuses to let one stand in for the other. The red-on-old, green-on-new rule is the part I would keep — a new test that passes on old code proves nothing, just as a duplicate webhook with a fresh timestamp proves nothing.
The case I am still unsure about is the plausible claim with no run behind it at all. What does your gate do when the agent describes a fix that looks right, the diff is tidy, and the tests pass — but the run it says it did never actually happened? Is there anything in the gate that ties the claim back to an execution record, or is that the next trust gap after the hidden suite?
That's a very good question. The gate should never accept "I ran the tests and they passed" as evidence by itself.
In my approach, "gate.sh" runs the checks independently against the committed code. The agent's explanation and its claimed test results don't determine the verdict. The gate's own execution result does.
So even if the diff looks clean and the agent describes a plausible fix, it still has to pass the checks when the gate runs them.
I agree there's another important layer here: making the execution record traceable to the exact commit, test suite, and gate run. That makes it possible to distinguish an actual verified execution from a log or claim supplied by the agent.
Your claim-row example is a useful parallel. In both cases, the system needs evidence of what actually happened, not just a plausible description of what happened.
I'd treat execution provenance as an important next step, especially for CI environments where results need to be independently auditable.
The red-on-old, green-on-new rule is the one I'd copy first. On amarinhodev's hardcoding case: if the hidden suite reuses the numbers in the bug report, an agent can pass it by special-casing those inputs. I'd write the hidden tests from the acceptance criterion, with fresh examples nobody pasted into the issue, and have the person who reported the bug confirm them. Then "1 fix claimed" can also name the criterion it claims to meet, and the gate checks that the hidden test for that criterion is the one that went green.
I like this approach. The acceptance criterion should be the source of truth, not the exact inputs from the bug report.
Using fresh examples makes it harder for an agent to pass by hardcoding the reported values. Having the person who reported the bug confirm the test also adds an important validation step: the hidden test must represent the intended behaviour, not just an assumption made by the test author.
I particularly like your suggestion to connect each claimed fix to a specific acceptance criterion. The verdict could then show which criterion was tested, which hidden test verified it, and whether that test passed against the original and updated code.
That gives us a much stronger audit trail than simply saying "1 fix claimed, 1 test passed."
Thank you for this, Sarvar. The "no model grades the model" rule is the strongest design decision here, and your build log shows why it matters: the two checks you added after the agent deleted a failing function and wrote a test that passed before and after the change are exactly the moves a tired reviewer approves at 5 p.m.
I especially liked the red-to-green requirement for new tests. Counting only the one proven test out of three, with two flagged as vacuous, turns "tests were added" from a comforting claim into evidence. Requiring that the agent's own test and the hidden suite agree before a fix counts as verified is a clean answer to the problem of one signal being gamed.
On your closing question, I'd say a hidden suite doesn't eliminate the trust problem so much as relocate it to something far more auditable: a suite a human wrote, versioned, and can inspect, rather than another model's opinion. The risks shift to coverage gaps (a fix can satisfy the hidden tests and still miss an edge case) and to suite leakage over many rounds, since feedback loops and best-of-3 sampling can slowly teach the agent what the suite rewards. Rotating or expanding the hidden tests periodically might help with that.
I also appreciated the honesty of your numbers, especially issue 4 converging in only 2 of 8 runs and the PR staying rejected when version 2 never converged. That's the gate working as intended. Excellent, practical work.
Thanks for the thoughtful feedback. You picked up on exactly what I wanted to demonstrate: adding tests is not the same as adding evidence that a fix works.
Your point about hidden-suite leakage is important. If an agent gets repeated feedback from the same suite, it may gradually learn what the suite rewards rather than fully solving the underlying problem. Versioning and periodically expanding the suite would help, but I'd also want to keep some tests completely undisclosed until final verification.
I also agree that coverage gaps remain. A green result means the change passed the checks we defined, not that every possible edge case is covered.
And yes, issue 4 staying rejected after version 2 failed to converge was intentional. The gate should report an unsuccessful fix rather than turn repeated attempts into an automatic pass.
I'm curious about one thing: would you rotate hidden tests between evaluation runs, or keep a stable core suite and add a smaller rotating set of tests around it?
"No model grades the model" is the rule I had to learn the hard way 🙌 My agents' summaries are claims too, and the scariest failure wasn't a broken test. A cheap model edited four JSON config files, listed exactly the changes I'd asked for in its summary, and had silently reverted a bunch of unrelated settings because it started from stale copies. Tests were green, the summary was clean.
What fixed it for me was boring: snapshot the files before the run, then diff by key afterwards and only merge the blocks that were asked for.
Does your gate also check scope, i.e. flag a PR that touches files or lines outside the fix it claims? That's the case I'd love to automate next 👀