In the first post a Scrum workflow for coding agents looked clean: with the skill, 6 of 6 runs met every item of the Definition of Done, against 3 of 6 without it. Several of you pointed out the obvious catch. One small task, one model, three runs per side. That's a hint, not a result.
So I ran a bigger one, and I wrote down the rules before I looked at a single run.
What changed
- A harder sprint. A new Go repo with four items that depend on each other across modules. Two of them carry an authorization rule the tickets only imply. One runs a code generator that also writes into a module no role may commit.
- Hidden acceptance tests. Each item has tests the agents never see. This is the part I care about most, because a tidy process that ships the wrong code is still the wrong code.
-
Seven models, two vendors. Haiku, Sonnet and Opus 5.5 in Claude Code;
gpt-6-luna,gpt-6.1-sol,gpt-5.6-solandgpt-6-astrain Codex CLI. Each with and without the skill. - Pre-registered. Hypotheses, measures, the rule for adding runs and the scoring script were frozen before the first data run. Every change after that is logged with a date.
Two runs per cell first. The pre-registered rule then flagged every model as inconsistent, so every cell got a third run. 42 runs, 14 hours, all on subscriptions.
What the skill still does
It makes the process complete. Without it, review verdicts and the sprint reports were missing in 10 of 21 runs. With it, 2 and 1 of 21. Same pattern as last time, now across seven models instead of one.
It also made runs cheaper for Sonnet, Opus and gpt-6-astra. I expected the opposite.
And nobody, in either arm, committed the generator's write into the other module. Strong models respect write regions when you spell them out.
What it doesn't do
It doesn't reliably make the code more correct. Hidden tests went up for Haiku (+1.3 of 4 items), Sonnet and gpt-5.6-sol. They stayed flat for gpt-6.1-sol and Astra, and dipped slightly for Opus and Luna. The whole hidden suite passed in 3 of 21 runs without the skill and 4 of 21 with it. A workflow can make sure tests exist and get reviewed. It can't make the reviewer spot an authorization rule nobody wrote down.
It doesn't help every small model. Haiku gained the most. Luna, the small Codex model, got worse with the skill. One of its runs merged nothing at all.
It can get expensive. With gpt-5.6-sol, the skill's portable mode burned through tokens: one attempt alone read 17.9 million input tokens. Cost per passed hidden item went from $0.68 to $2.79.
The bug in my own grader
This one stung. One Definition-of-Done item checks that implementation and tests were committed separately for each item. My grader found an item's commits through its branch or an item ID in the commit message. Lanewright deletes item branches after merging, and its role commits don't repeat the ID. So the grader saw only the docs commits and failed that item in 17 of 21 runs with the skill.
The separation was there. Reading the same commits from each merged pull request, with the same rules, gives 19 of 21 with the skill and 21 of 21 without.
I didn't touch the frozen grader. The report shows its numbers as the main result and the re-scored ones next to them. With the fix, the skill raised the Definition-of-Done score in 6 of 7 models. Without it, in 3. You can decide which one you believe; both are in the data.
Claude vs Codex, briefly
Matched by list price, the Claude model met as many or more items in 7 of 8 comparisons. Codex was cheaper per passed item on the cheap and middle pairs. On the expensive pairs it flipped. Price isn't ability and the model generations differ, so I'd read this as a sketch, not a ranking.
What's next
Sixteen more runs start this weekend: Lanewright against the free scrum-kit in Claude Code, across four model settings, including a cheap orchestrator driving stronger roles. After that the task, hidden tests and harness go public too.
Everything above, with every run's numbers, the pre-registration and the deviations log, is here: bench/hard-2026-11-10.
If you want to try it
Lanewright is a paid skill on Agensi: Lanewright on Agensi. Version 1.0.1, the exact build tested here.
The planning side is free. Four MIT skills take a loose brief to an SRS, a weekly sprint plan and a backlog a human signs off before any agent starts:
Intake,
SRS,
Sprint Planner,
Backlog Check.
And scrum-kit stays free and MIT, with prompts in Bahasa Indonesia.
Correction, 15 Nov 2026. Two Codex runs without the skill found the claude CLI installed on the same machine and handed every role to Claude Code through claude -p. Their Claude sessions weren't in the recorded cost. I declared both runs invalid and reran them with claude hidden from Codex. Two things above changed: the skill no longer makes gpt-6.1-sol runs cheaper (now $0.73 with it against $0.66 without), and for gpt-5.6-sol the cost per passed item without the skill is $0.68, not $0.72. Everything else holds. Details and the old numbers: https://github.com/huzairsaputra/lanewright/blob/main/bench/hard-2026-11-10/README.md

Top comments (3)
Correction for anyone who read this early. Two of the Codex runs without the skill found Claude Code installed on my machine and quietly handed every role to it, so their Claude cost never showed up in my numbers. I reran both with claude hidden from Codex. The skill now makes gpt-6.1-sol slightly more expensive instead of slightly cheaper, and the rest of the post holds. The same check found a rule in my own pack that could let a Codex orchestrator start Claude roles, and that is fixed for 1.1.0. Details here github.com/huzairsaputra/lanewrigh...
One more follow-up. A few people asked whether Claude Code's advisor tool would rescue the cheap Haiku orchestrator, so I tested it. Twelve more sprints with an Opus 5.5 advisor behind a Haiku or a Sonnet orchestrator, Lanewright and scrum-kit, three runs each.
Short answer, no. The advisor was called 8 times in 12 runs, always by the orchestrator, always in the first two minutes while it planned, never by a role and never at the point where a run went wrong. Haiku with the advisor passed 1.7 and 2.0 of 4 hidden items against 1.5 and 2.5 without it, and the advisor was 11 to 16 percent of the cost of those runs. Sonnet without an advisor stayed the cheapest per passed item, $0.63 and $0.81 with 4 of 4 hidden items.
The task, hidden tests, grader, prompts and harness are public now, so anyone can rerun the benchmark with their own pack or none. Paid Lanewright isn't in it. The Claude Code and Codex runs from this post are also regrouped by price pair on one page.
Advisor write-up with every run and the deviations github.com/huzairsaputra/lanewrigh...
Kit github.com/huzairsaputra/lanewrigh...
Claude against Codex github.com/huzairsaputra/lanewrigh...
Follow-up for anyone who read this far. The extra phase is done and published.
Sixteen more sprints in Claude Code, Lanewright against the free scrum-kit, four model settings, two runs each. Lanewright met more Definition-of-Done items in all four settings and passed as many hidden tests or more. scrum-kit was cheaper per run in all four, by 16 to 29 percent. A Sonnet orchestrator with Opus roles beat all-Opus in both packs, with more hidden tests passing and a lower cost per passed item. A Haiku orchestrator was the least stable setting in both.
One correction too. My harness priced Sonnet cache reads at twice the list price, so the Sonnet costs in this post were about 12 to 25 percent too high. Nothing in the conclusions flips, and the corrected numbers are in the repo with a dated note.
Every run, the charts and the deviations log are here github.com/huzairsaputra/lanewrigh...