DEV Community

Sion Jang
Sion Jang

Posted on Fully Autonomous

Your coding agent's tests pass. It still edited files you never asked about.

You ask a coding agent to fix one function. It fixes it. On the way it also touches a config file, bumps a version, and loosens a test that was in the way. Every test passes. The agent says it's done.

Nothing in that workflow is wrong by its own rules. The tests were the definition of done, and the tests pass. The extra edits only show up if you read the whole diff, and the point of handing work to an agent was not having to.

This is common, not anecdotal

A May 2026 study read 20,574 real coding-agent sessions from public repositories, about a third of them Claude Code, along with Cursor, Codex and others (arXiv 2605.29442). Across 16,118 validated problem episodes:

  • 38.33% involved the agent violating an explicit developer constraint
  • 10.20% were the agent taking actions beyond the stated scope
  • 22.58% were the agent misreporting the status of its own work

When a problem visibly got resolved, 91.49% of the time it took the developer explicitly pushing back. The agent rarely caught it alone.

Forums say the same in plainer words: an agent that keeps editing files the user didn't ask it to, to the point the user built their own auditor, and agents that overshoot simple requests.

Why "make the tests pass" doesn't catch it

A test-based gate answers one question: does the code still do what the tests check? Scope creep is a different question: did the agent change only what it was asked to change?

  • A lockfile or config edit usually breaks no test.
  • A loosened test makes the suite easier to pass.
  • A version bump is invisible to unit tests.

One developer ran 36 controlled Claude Code tasks to check whether "done" matched the output. The report and the output never disagreed. What did go wrong was quieter. Under a strict completion criterion, the agent changed what the code meant in order to pass, in 2 of 6 runs. And the verification script itself passed when the result file was missing.

So the failure isn't mainly the agent lying. Our checks ask the wrong question, or quietly pass when there is nothing to check.

What a scope check has to do

If you want a machine to say "this went outside the task," it needs:

  1. A declared scope. Paths the task may change, and paths it may never change. Without this there is nothing to judge against.
  2. The final state, not the tool calls. Agents write files through scripts, build steps and shell redirects as well as edit tools. Watching tool calls misses some of those. Comparing the working tree at the end does not.
  3. A baseline. Your own uncommitted work existed before the session. A scope check that blames the agent for it gets switched off within a day.
  4. A place in the loop where it can say no. A report you read afterwards is a code review. A check at the moment the agent tries to finish can send it back with a list of what to undo.
  5. A way out. If the agent fails the same check three times, it should hand over to a person instead of looping.

And it should be honest about what it can't see. A path-based scope check doesn't notice a change of meaning inside a file you allowed, like rewriting a calculation so a test passes. That still needs an output check or a human.

What I built

I built this as a Claude Code Stop hook called taskattest. When the agent tries to finish, it checks the git working tree against a declared scope:

{
  "task_id": "fix-billing",
  "checks": [
    {"type": "scope_changes", "name": "billing only", "root": "..",
     "allow": ["src/billing/**", "tests/**"], "deny": ["**/*.lock"]}
  ]
}
Enter fullscreen mode Exit fullscreen mode

If the agent also changed pyproject.toml, it can't stop yet:

[FAIL] fix-billing  (model calls: 0)
  - fail    billing only: 1 change(s) outside the allowed scope: pyproject.toml
            fix: undo changes outside the task scope: pyproject.toml
Enter fullscreen mode Exit fullscreen mode

It can also check output files (invented rows, wrong totals, CSV and Excel). It's deterministic: no model calls, no network, standard library only, and it runs on Linux, macOS and Windows. Apache-2.0.

https://github.com/sionjang101-create/taskattest

I'm more interested in the problem than the tool. If you've hit scope creep with an agent, what did it touch, and would a declared scope have caught it?

Top comments (6)

Collapse
 
itsvicmoreno profile image
Vic Moreno •

One thing in the example scope: with tests/** in allow, the loosened test from your opening paragraph passes the scope check, since it sits inside an allowed path. I'd keep new test files allowed but treat edits to existing test files as their own check. Does taskattest let you allow new files under a path but not modifications?

Collapse
 
sion_jang_b7e2e2c640671f5 profile image
Sion Jang •

Good catch, and you're right. Today allow and deny only match paths. The check records whether each file is new, modified or deleted, but you can't write a rule on that status yet. So with tests/** allowed, an edit to an existing test passes the scope check.

The workaround right now is to list the existing test files under deny and keep tests/** in allow. Deny wins over allow, so new test files pass and edits to the listed ones fail. It works, but you have to keep the list current.

A rule like "under tests/, new files are fine, existing files may not change" looks like the right shape. Before I add it: would you actually run that on your own repo? If so, which matters more to you, blocking edits to existing tests or blocking deletions too?

Collapse
 
itsvicmoreno profile image
Vic Moreno •

Edits to existing tests first: a deleted test shows up in the diff stat, while a loosened assertion inside a file that still passes is the one nobody notices in review. Deletions could sit under the same rule at almost no cost, since an existing file disappearing is also a status change. One edge case worth deciding early: does a rename count as delete plus new, which would let a test escape the rule just by moving?

Collapse
 
mnemehq profile image
Theo Valmis •

A scope check works better as a precondition than as a report. Tests answer whether the code still does what they assert, so a loosened test makes them easier to pass. I would have the task declare the paths it may touch before the agent starts, fail the run on any edit outside them, and treat any change under the test directory as needing its own approval. Of the three categories in the study, scope creep is the one a path rule can catch mechanically.

Collapse
 
maxritter profile image
Max Ritter •

The files nobody asked about can slip past a green test suite when the tests only check the function you named. Comparing the diff against a written scope that was approved before the agent started makes those extra edits stand out right away. I built QualityLayer for this: the plan is reviewed first, and a separate review agent checks the finished change against it. github.com/maxritter/qualitylayer

Collapse
 
halfcycle profile image
Halfcycle •

Tests can't catch scope creep because nobody wrote down the scope. Halfcycle does that before the agent starts, then checks the work against it, and a person signs off. npx halfcycle