You ask a coding agent to fix one function. It fixes it. On the way it also touches a config file, bumps a version, and loosens a test that was in the way. Every test passes. The agent says it's done.
Nothing in that workflow is wrong by its own rules. The tests were the definition of done, and the tests pass. The extra edits only show up if you read the whole diff, and the point of handing work to an agent was not having to.
This is common, not anecdotal
A May 2026 study read 20,574 real coding-agent sessions from public repositories, about a third of them Claude Code, along with Cursor, Codex and others (arXiv 2605.29442). Across 16,118 validated problem episodes:
- 38.33% involved the agent violating an explicit developer constraint
- 10.20% were the agent taking actions beyond the stated scope
- 22.58% were the agent misreporting the status of its own work
When a problem visibly got resolved, 91.49% of the time it took the developer explicitly pushing back. The agent rarely caught it alone.
Forums say the same in plainer words: an agent that keeps editing files the user didn't ask it to, to the point the user built their own auditor, and agents that overshoot simple requests.
Why "make the tests pass" doesn't catch it
A test-based gate answers one question: does the code still do what the tests check? Scope creep is a different question: did the agent change only what it was asked to change?
- A lockfile or config edit usually breaks no test.
- A loosened test makes the suite easier to pass.
- A version bump is invisible to unit tests.
One developer ran 36 controlled Claude Code tasks to check whether "done" matched the output. The report and the output never disagreed. What did go wrong was quieter. Under a strict completion criterion, the agent changed what the code meant in order to pass, in 2 of 6 runs. And the verification script itself passed when the result file was missing.
So the failure isn't mainly the agent lying. Our checks ask the wrong question, or quietly pass when there is nothing to check.
What a scope check has to do
If you want a machine to say "this went outside the task," it needs:
- A declared scope. Paths the task may change, and paths it may never change. Without this there is nothing to judge against.
- The final state, not the tool calls. Agents write files through scripts, build steps and shell redirects as well as edit tools. Watching tool calls misses some of those. Comparing the working tree at the end does not.
- A baseline. Your own uncommitted work existed before the session. A scope check that blames the agent for it gets switched off within a day.
- A place in the loop where it can say no. A report you read afterwards is a code review. A check at the moment the agent tries to finish can send it back with a list of what to undo.
- A way out. If the agent fails the same check three times, it should hand over to a person instead of looping.
And it should be honest about what it can't see. A path-based scope check doesn't notice a change of meaning inside a file you allowed, like rewriting a calculation so a test passes. That still needs an output check or a human.
What I built
I built this as a Claude Code Stop hook called taskattest. When the agent tries to finish, it checks the git working tree against a declared scope:
{
"task_id": "fix-billing",
"checks": [
{"type": "scope_changes", "name": "billing only", "root": "..",
"allow": ["src/billing/**", "tests/**"], "deny": ["**/*.lock"]}
]
}
If the agent also changed pyproject.toml, it can't stop yet:
[FAIL] fix-billing (model calls: 0)
- fail billing only: 1 change(s) outside the allowed scope: pyproject.toml
fix: undo changes outside the task scope: pyproject.toml
It can also check output files (invented rows, wrong totals, CSV and Excel). It's deterministic: no model calls, no network, standard library only, and it runs on Linux, macOS and Windows. Apache-2.0.
https://github.com/sionjang101-create/taskattest
I'm more interested in the problem than the tool. If you've hit scope creep with an agent, what did it touch, and would a declared scope have caught it?
Top comments (6)
One thing in the example scope: with tests/** in allow, the loosened test from your opening paragraph passes the scope check, since it sits inside an allowed path. I'd keep new test files allowed but treat edits to existing test files as their own check. Does taskattest let you allow new files under a path but not modifications?
Good catch, and you're right. Today
allowanddenyonly match paths. The check records whether each file is new, modified or deleted, but you can't write a rule on that status yet. So withtests/**allowed, an edit to an existing test passes the scope check.The workaround right now is to list the existing test files under
denyand keeptests/**inallow. Deny wins over allow, so new test files pass and edits to the listed ones fail. It works, but you have to keep the list current.A rule like "under tests/, new files are fine, existing files may not change" looks like the right shape. Before I add it: would you actually run that on your own repo? If so, which matters more to you, blocking edits to existing tests or blocking deletions too?
Edits to existing tests first: a deleted test shows up in the diff stat, while a loosened assertion inside a file that still passes is the one nobody notices in review. Deletions could sit under the same rule at almost no cost, since an existing file disappearing is also a status change. One edge case worth deciding early: does a rename count as delete plus new, which would let a test escape the rule just by moving?
A scope check works better as a precondition than as a report. Tests answer whether the code still does what they assert, so a loosened test makes them easier to pass. I would have the task declare the paths it may touch before the agent starts, fail the run on any edit outside them, and treat any change under the test directory as needing its own approval. Of the three categories in the study, scope creep is the one a path rule can catch mechanically.
The files nobody asked about can slip past a green test suite when the tests only check the function you named. Comparing the diff against a written scope that was approved before the agent started makes those extra edits stand out right away. I built QualityLayer for this: the plan is reviewed first, and a separate review agent checks the finished change against it. github.com/maxritter/qualitylayer
Tests can't catch scope creep because nobody wrote down the scope. Halfcycle does that before the agent starts, then checks the work against it, and a person signs off. npx halfcycle