For the harder-task run, one thing worth measuring: green CI on each PR's head says nothing about the PRs together. Our team of Claude Code agents hit exactly that on Sunday: 9 backend PRs that were each fine conflicted with each other. One agent built a single integration branch (merge each PR, no rebase), a ChatGPT agent reviewed only the conflict resolutions, and a full CI run on the merged result found 5 failures no single PR showed. Since then only one agent merges into staging, one PR at a time, never while a pipeline is running. The coordination around it (who merges next, which PR waits for which) happens in agent-chat, the tool we build, where the agents mention each other and the reviewer answers in the same room. Does Lanewright's integration PR get its own CI run on the combined state, or only per item?
Good question, and fair. Yes, it’s two layers. Every item goes into the integration branch through its own PR, and only after CI is green on that PR’s head commit. At the end of the sprint there’s one draft PR from the integration branch to the base branch, and the CI result on its head commit goes into the description. So the combined state does get a full CI run before a human approves the merge.
The limit is simple though. If your tests don’t cover how items interact, the combined run won’t catch it either. Your nine PRs are exactly why the merge to the base still needs a person to sign off.
@huzair_saputra Agreed on the split: a missed Definition-of-Done item counts as a failed run here, and no broken code was merged. Harder tasks are the test that decides whether the extra roles earn their cost.
@huzair_saputra 6 of 6 against 3 of 6 is the number I would have asked for first. Every miss was a report or the sprint summary left uncommitted, not broken code, so the roles bought completeness. At 1.5x the cost, that is the number to weigh, and your failed-run count is the one still to measure.
It depends on how we define a "failed run":
• If it means missing a 'Definition of Done' item: Without the pack, 3 out of 6 runs failed (due to uncommitted reports/summaries). With the pack, 0 out of 6 failed.
• If it means broken work/code: Neither side produced a failed run. All 12 runs merged successfully via PR after CI went green, and no one wrote outside their region. The task was likely too easy to show if extra roles prevent these failures.
For broader context, across 22 published runs with the pack: 20 finished successfully and 2 safely paused for human decision.
I’ve detailed the full tally and its limitations in the repo. Moving forward, I plan to test this on harder tasks with more runs and lock down the exact definition of a "failed run" before analyzing the results.
Two agents on one file is the failure to fix first. The number to read is whether the extra roles cut failed runs enough to pay for themselves. Fewer failed runs would be the proof, not a longer chain.
Fair point, and I'd rather agree than defend the chain. From my data, including where it doesn't prove your point yet:
Two agents on one file: roles work in separate worktrees and each has a write region, enforced by deny rules in Claude Code and by a check before every commit elsewhere. In the benchmark no commit landed outside its region, with or without the skill. So the failure never happened in either arm; the task couldn't tell them apart. That is not proof the enforcement works.
Failed runs: I can't claim what you're asking for. When the Definition of Done is written out, a strong model already follows most of it: all 26 merges waited for green CI on the PR head, in all 12 runs. The difference was completeness, 6 of 6 runs with the skill against 3 of 6 without, and every miss was reports and the sprint summary not being committed, not broken code. The skill costs about 1.5x more and 1.4x the time in Claude Code. So the extra roles made the process complete, but I have no evidence yet that they cut failed runs.
What would count as proof, as you say: fewer runs failing hidden acceptance tests per task, with and without the roles, on far more than 3 runs per arm. I'd like to run that. Until then please read my numbers as the cost of the extra roles, not as payback.
The useful distinction here is that the skill seems to improve completeness and consistency, not make agents follow a process they would otherwise ignore; the fixed Definition of Done and PR-head CI checks make that a much clearer result. The Codex limitation is especially important: a pre-commit ownership check can catch a boundary violation, but it cannot prevent the work or side effects beforehand. Did you track how often Codex crossed a region before the check caught it?
Thanks, and you're right about the limit: on Codex the check reports rather than blocks, so it only sees the result.
To answer directly: no, I didn't track attempts. I scored the final state from git and the GitHub API, and the number there is 0 commits outside a region in all 12 runs, Codex included. That tells you nothing about how often an agent touched a file it shouldn't have and then reverted it, or about side effects that never reach a commit (a generator rewriting files, for example; the BE-2 task has one on purpose). The check also lists untracked and staged files, so it catches a crossing that was never committed, but I didn't log how many times it fired.
That is the gap I'd close next: record every failed ownership check per run, per agent, so "how often does it cross before the check" becomes a number instead of a guess.
The rule that the reviewer wrote neither the code nor the tests is the piece I'd keep even without the rest of the process. Is there anything that checks the backlog item itself before the implementer picks it up? That's the gap I've been working on with Open Council: the seats argue the spec out to sign-off and a human approves it before any role touches the repo. It would sit in front of something like this rather than replace it. github.com/goshtasb/OpenCouncil
Agreed on the reviewer rule; it's the part I'd keep too.
On checking the item before it's picked up: yes, a smaller version of that exists. The sprint tools stop at a readiness gate. A separate free skill, Backlog Check, reads the backlog against the SRS (acceptance criteria present and testable, story points, every item traced to a requirement, placeholders and vague wording flagged) and writes a report with a verdict, a SHA-256 fingerprint of the backlog and a sign-off block that only a human fills. If the report is missing, stale (backlog edited after the check), not ready, or unsigned, the sprint asks before starting. It's mostly mechanical checks plus a checklist, not a debate between roles.
So I'd say it's the opposite shape to yours: mine checks that a written backlog is ready, yours argues the spec out before there is one. They look complementary. I haven't tried Open Council, so I can't compare them beyond that.
For the harder-task run, one thing worth measuring: green CI on each PR's head says nothing about the PRs together. Our team of Claude Code agents hit exactly that on Sunday: 9 backend PRs that were each fine conflicted with each other. One agent built a single integration branch (merge each PR, no rebase), a ChatGPT agent reviewed only the conflict resolutions, and a full CI run on the merged result found 5 failures no single PR showed. Since then only one agent merges into staging, one PR at a time, never while a pipeline is running. The coordination around it (who merges next, which PR waits for which) happens in agent-chat, the tool we build, where the agents mention each other and the reviewer answers in the same room. Does Lanewright's integration PR get its own CI run on the combined state, or only per item?
Good question, and fair. Yes, it’s two layers. Every item goes into the integration branch through its own PR, and only after CI is green on that PR’s head commit. At the end of the sprint there’s one draft PR from the integration branch to the base branch, and the CI result on its head commit goes into the description. So the combined state does get a full CI run before a human approves the merge.
The limit is simple though. If your tests don’t cover how items interact, the combined run won’t catch it either. Your nine PRs are exactly why the merge to the base still needs a person to sign off.
@huzair_saputra Agreed on the split: a missed Definition-of-Done item counts as a failed run here, and no broken code was merged. Harder tasks are the test that decides whether the extra roles earn their cost.
Thanks, agreed on all of it. The harder task is the next test I want to run properly, and I will publish the result either way.
@huzair_saputra 6 of 6 against 3 of 6 is the number I would have asked for first. Every miss was a report or the sprint summary left uncommitted, not broken code, so the roles bought completeness. At 1.5x the cost, that is the number to weigh, and your failed-run count is the one still to measure.
It depends on how we define a "failed run":
• If it means missing a 'Definition of Done' item: Without the pack, 3 out of 6 runs failed (due to uncommitted reports/summaries). With the pack, 0 out of 6 failed.
• If it means broken work/code: Neither side produced a failed run. All 12 runs merged successfully via PR after CI went green, and no one wrote outside their region. The task was likely too easy to show if extra roles prevent these failures.
For broader context, across 22 published runs with the pack: 20 finished successfully and 2 safely paused for human decision.
I’ve detailed the full tally and its limitations in the repo. Moving forward, I plan to test this on harder tasks with more runs and lock down the exact definition of a "failed run" before analyzing the results.
Two agents on one file is the failure to fix first. The number to read is whether the extra roles cut failed runs enough to pay for themselves. Fewer failed runs would be the proof, not a longer chain.
Fair point, and I'd rather agree than defend the chain. From my data, including where it doesn't prove your point yet:
Two agents on one file: roles work in separate worktrees and each has a write region, enforced by deny rules in Claude Code and by a check before every commit elsewhere. In the benchmark no commit landed outside its region, with or without the skill. So the failure never happened in either arm; the task couldn't tell them apart. That is not proof the enforcement works.
Failed runs: I can't claim what you're asking for. When the Definition of Done is written out, a strong model already follows most of it: all 26 merges waited for green CI on the PR head, in all 12 runs. The difference was completeness, 6 of 6 runs with the skill against 3 of 6 without, and every miss was reports and the sprint summary not being committed, not broken code. The skill costs about 1.5x more and 1.4x the time in Claude Code. So the extra roles made the process complete, but I have no evidence yet that they cut failed runs.
What would count as proof, as you say: fewer runs failing hidden acceptance tests per task, with and without the roles, on far more than 3 runs per arm. I'd like to run that. Until then please read my numbers as the cost of the extra roles, not as payback.
Raw data and the scoring script are public, so you can check them: github.com/huzairsaputra/lanewright (bench/fair-2026-11-06)
The useful distinction here is that the skill seems to improve completeness and consistency, not make agents follow a process they would otherwise ignore; the fixed Definition of Done and PR-head CI checks make that a much clearer result. The Codex limitation is especially important: a pre-commit ownership check can catch a boundary violation, but it cannot prevent the work or side effects beforehand. Did you track how often Codex crossed a region before the check caught it?
Thanks, and you're right about the limit: on Codex the check reports rather than blocks, so it only sees the result.
To answer directly: no, I didn't track attempts. I scored the final state from git and the GitHub API, and the number there is 0 commits outside a region in all 12 runs, Codex included. That tells you nothing about how often an agent touched a file it shouldn't have and then reverted it, or about side effects that never reach a commit (a generator rewriting files, for example; the BE-2 task has one on purpose). The check also lists untracked and staged files, so it catches a crossing that was never committed, but I didn't log how many times it fired.
That is the gap I'd close next: record every failed ownership check per run, per agent, so "how often does it cross before the check" becomes a number instead of a guess.
The rule that the reviewer wrote neither the code nor the tests is the piece I'd keep even without the rest of the process. Is there anything that checks the backlog item itself before the implementer picks it up? That's the gap I've been working on with Open Council: the seats argue the spec out to sign-off and a human approves it before any role touches the repo. It would sit in front of something like this rather than replace it. github.com/goshtasb/OpenCouncil
Agreed on the reviewer rule; it's the part I'd keep too.
On checking the item before it's picked up: yes, a smaller version of that exists. The sprint tools stop at a readiness gate. A separate free skill, Backlog Check, reads the backlog against the SRS (acceptance criteria present and testable, story points, every item traced to a requirement, placeholders and vague wording flagged) and writes a report with a verdict, a SHA-256 fingerprint of the backlog and a sign-off block that only a human fills. If the report is missing, stale (backlog edited after the check), not ready, or unsigned, the sprint asks before starting. It's mostly mechanical checks plus a checklist, not a debate between roles.
So I'd say it's the opposite shape to yours: mine checks that a written backlog is ready, yours argues the spec out before there is one. They look complementary. I haven't tried Open Council, so I can't compare them beyond that.
Backlog Check lives here: github.com/huzairsaputra/lanewrigh...