This week a lot of people on X argued about "software factories" for AI coding. Some companies sell pipelines of agents, and their critics say most of it is marketing and the parts that work are the deterministic ones.
We run something close to a factory. Coding agents write most of the code for our TypeScript framework, Kumiko, and the apps built on it, and an automated review has read 5,322 of our merged PRs since May 27. The rules for the agents are about 3,100 lines of Markdown in 32 files. I went through our agent transcripts to see which of those rules held. The critics are right, but "deterministic" covers less than I thought.
A note didn't hold
On November 1 we wrote down that the main agent starts coding agents itself, with no Opus agent in between that only passes the prompt along. Between then and the evening of November 5, 22 general-purpose agents started a coding agent anyway, 53 times in total.
That evening we added a hook that refuses the spawn when a subagent tries it. It has fired once since, in our own test. In the two days after, coding agents were started only by the main agent or by the one exception we allow.
So the hook can't take the credit, at least not yet. The same commit rewrote the rule, and something the hook can't touch changed too: the main agent used to hand work to a general-purpose agent about 20 times a day, and since then it's about 3 times a day. The hook only checks spawns from subagents. The drop came from the rewritten Markdown, or from a different mix of work in those two days.
A hook didn't hold either
Our pre-push hook runs typecheck, lint, unit tests and our own guards. Skipping it with --no-verify or PRE_PUSH_SKIP=1 is on the short list of hard bans in our agent config, with one exception: the failure is outside your diff, you can show who caused it, and you ran the skipped checks on your own files.
Since October 20 our transcripts contain 411 push commands, and 118 of them skipped the hook.
| Repo | Push commands | Skipped the hook |
|---|---|---|
| Infrastructure | 104 | 69 |
| Framework | 135 | 32 |
| One of the apps | 57 | 1 |
| Everything else | 115 | 16 |
I read the reasons the agents wrote next to the skips. Most point to red outside their diff, often with the three proofs listed. Some name tests that time out when the machine is busy, and the exception explicitly excludes flaky tests. Some give no reason at all.
The infrastructure repo explains most of the skips. Its hook also scans the neighbouring repos, so it goes red on code the agent never touched. When a gate keeps failing for someone else's change, the exception gets used on every push, and after a while that's simply how pushing works in that repo. A memory note our agents keep for themselves even called PRE_PUSH_SKIP=1 the documented pattern. I corrected it today.
What held
The checks on GitHub. In every repo those pushes went to, main only takes a merge after the required checks pass, and the rule applies to admins too. All 118 skipped pushes still had to get through CI before anything merged. An agent on a laptop can't skip a check that runs somewhere else.
Our cast audit shows the same thing. It counts suspicious as casts in the framework against a frozen baseline and fails when a change adds one. It runs inside the same pre-push hook, so it got skipped along with everything else, but CI runs it too. The baseline stood at 349 in June, after we extended the audit's coverage, and it's at 69 now.
What I'd tell someone building one
Put the check that must hold on the server. Keep the local hook fast, and let it look only at the diff being pushed, or it will cry wolf until nobody listens. Write the Markdown anyway, as the reason behind a check, but don't count on it to be the check.
Speed matters for the same reason. When we moved our per-file guards into Biome plugins on November 4, lint got two to three times slower. A slower hook is one more reason to skip it, and we haven't fixed that yet.
What this doesn't prove
The numbers come from our agent transcripts in this workspace, counted per tool call. A push that was retried counts twice.
The reasons come from the short notes the agents wrote next to each push. I didn't replay the sessions, so I can't say how many of the claimed three proofs would hold up.
The spawn hook has two days of data and hasn't been triggered by real work yet.
We didn't compare bug rates before and after any of this, so I can't say what the skipped hooks cost us or how much CI caught in their place.
A question
If you run agents on your repos, which check do you treat as the one that can't be skipped? And how do you keep the local one from going red on code the agent didn't touch?
Top comments (4)
The hook can't touch the main agent's handoffs, so the drop from about 20 a day to about 3 is really a test of the rewritten Markdown against a different mix of work. If you already tag tasks (refactor, test fix, new feature), compare the handoff rate within each type before and after November 5. If it also falls inside refactors alone, the mix can't be the whole explanation.
Fair point, and we can't rule it out. We don't tag tasks by type. The closest thing is the one-line description each spawn gets, and that's free text. November 5 is also only three days back, so a split by type would leave a handful of spawns per bucket. If the drop holds for a few weeks, splitting by repo is probably the cleanest cut we have, since infrastructure work and app work look very different.
Infrastructure skipping 69 of 104 next to one app at 1 of 57 suggests the skips track how often things are red outside the diff in that repo more than agent discipline. If most written reasons point to failures outside the diff, the exception has quietly become the normal path. Checking each stated reason against what actually failed would show how many of the three proofs really hold. Did you find skips where the stated cause didn't match the real failure?
Yes, at least twice that we know of. On October 30 the infrastructure agent skipped the hook on three pushes because tests failed on code from another repo. The real cause was a missing
bun installin its fresh worktree, so the tests picked up our framework from the local checkout. One install and they passed. On November 2 the same thing happened in the framework repo.So the second proof, that someone else caused the failure, is the one that breaks most easily, and nothing checks it after the push. You're right about infrastructure too: its hook also scans the neighbouring repos, so red anywhere stops it. I haven't gone through all 118 reasons, so I can't give you a rate.