Your last bullet, screenshots by default, is a rule in our release process at work. Every feature attaches two short videos before it reaches QA: one showing the product, one walking through the interfaces and the contract. The product owner and the dev lead check those first, then QA tests from the feature doc, the way a client would hit it, not from the diff. Did October's fixes cluster in the screens, or in the parts you trust the tests for?
Mostly in the parts I trust the tests for. Counting October PR titles that start with "fix", about 45% of the API's merged PRs were fixes and about 47% of the frontend's, so neither repo stands out, and the API had nearly twice as many in total. Of the frontend fixes, at most about a quarter look like screen problems by their titles: layout, scrolling, wrapping, mobile.
Almost every fix in both repos also touched a test file. I read that as the agent adding a regression test along with the fix, not the tests catching the bug beforehand. This is a rough count from titles, not a look at each PR.
Your two videos are close to what I said I should have done with screenshots from the start, just with the contract walkthrough added.
A regression test that lands with the fix records where the bug was, not who caught it. That is why our QA writes its cases from the feature doc at hand-off, while the dev builds, not after. How many of the October fixes started as a report from a user rather than your own review?
Almost none, going by the issues. Most of October's 278 fixes point back at an issue, 209 different ones. By the issue text, 20 started from Sentry, half of them through a button on our Discord alerts that files the issue straight from the error. 8 were bugs I reported myself, and 1 came from a player. The other 180 or so read like the agent's own findings from reading the code, and a lot of them are follow-ups to an earlier fix. I've only read a sample of those, so take that split as rough.
Which supports your point. The regression test says where the bug was. It took the issue history to say who found it, and by that count it was mostly the agent, not users and not me.
180 of 278 from the agent's own reading is more than I would have guessed. The follow-ups are the part I would count next. My experience is that agents are not deterministic: one run follows every rule, the next does not, so a fix can need a second fix. How soon after the first fix did the follow-ups usually land?
Quickly, for most of them. I checked it the git blame way: for each October fix, which earlier PR last wrote the lines it changed. For 82 of 236 fixes, that earlier PR was itself a fix. Half of those follow-ups landed within about 4 days of the first one. 23 came within a day and 50 within a week. The rest are spread out, some months later, and those are probably not a second try at the same bug.
Blame only shows who last touched the lines, not that it was the same bug, so it's a rough count in both directions. But it does fit what you describe. When a fix needs another fix, it mostly shows up in the same week.
Visibility is a good compression strategy, but the ranking signal needs a falsifiable link to risk. Diff size alone is noisy; boundary crossings, permission changes, generated-file churn, and behavior not covered by tests are often better review triggers. I’d preserve the reason a PR was surfaced alongside the PR, then audit false negatives after incidents to tune the queue.
Creator of Semitexa, an open-source PHP framework built for AI agents. 18+ years in PHP. Writing about agents that follow the architecture instead of breaking it.
The line that stuck with me is "read the description like it's the contract". I agree, and it's also the one input on your list that nothing checks against the diff. CI checks the code, you check the description against what you asked for, but the link between description and diff is taken on trust.
We had exactly that case. While we were testing a GitHub Action in our own test app, Claude Code made a commit titled "Limit contact name to 250 chars". The sed it ran hit the email rule instead, because the name was already max:200. Tests green, message perfectly sensible, wrong line changed. Since then we run a check that compares what the message claims with what the diff touches. On 937 commits from five PHP repos it flagged 23, about 2.5%, so it's quiet enough to leave on.
With descriptions at 2,400+ characters there's even more room for the text to drift from the code. Have you caught one of your 1,396 where the description was right about the intent and wrong about what actually changed?
Not that I can show you, because I haven't run anything like your check. I tried the cheap version on October's PRs, matching the scope in the title against the paths the diff touches, and it's useless here. 107 of 520 scoped titles didn't match any path, and the ones I went through were all feature names rather than folders: "grid rooms", "tenagoal", "guesses". The scope is how I think about the product, not where the code lives.
So a check like yours would have to read the claim and the diff, the way yours does, not compare two strings. Your sed example is exactly the case my setup wouldn't catch: tests green, description sensible, wrong line. 2.5% flagged on 937 commits is quiet enough to be worth trying.
Creator of Semitexa, an open-source PHP framework built for AI agents. 18+ years in PHP. Writing about agents that follow the architecture instead of breaking it.
Feature names instead of folders is the part that kills any string comparison, and it's the same reason a path check couldn't have caught the sed case: the title said the right thing about the right feature. The line that was wrong was simply not the one the sentence was about.
That's why our check starts from the claim, not from the scope. It takes what the commit message says it changed and looks for the changed lines that sentence cannot account for. On the 937 commits that gave 23 flags, and the ones I read were all message and diff disagreeing, not naming conventions.
If you try it on October's 520, the useful number is not the flag rate but how many flags you'd have wanted to see. I'd treat anything above a handful a week as a bug in the check, not in your PRs.
QA Team Lead with a background in software engineering, startups, and technical leadership. Passionate about test automation, AI, and building high-quality software.
Interesting perspective! The increase in fix-related PRs is particularly telling. As AI increases development speed, I think QA becomes even more important. Green CI is valuable, but we also need confidence that we're testing the right things, not just that the tests pass.
I research AI tools, APIs & agents — comparing specs, pricing and real user reports — then publish honest rankings so you don't waste money. Research-first, never sponsored fluff.
1,241 agent PRs and still opening the ones you can actually look at — that's the honest version of review, not the pretend one. I do the same with screenshots and pages, and wave the rest through on green CI with a straight face. Did anything in the ones you did look at ever make you go back and re-check a merged one you hadn't?
The history can't say what prompted a second look, but it does show how often a merged PR got revisited. I blamed the lines each October fix changed back to the PR that last touched them. 81 of 235 fixes went back to something merged in the previous seven days, and 82 repaired a PR that was itself a fix.
So going back happens a lot. It just happens as another PR instead of a revert, which is why the revert count since April is still zero. Whether a preview or a Sentry alert started it is not something I can pull apart from the data. For the fixes that mention Sentry, the median gap from the broken PR to its fix was 14 days, against 41 for fixes overall, so the alerts at least get there sooner.
I write when an idea won't leave me alone 🧠 Building AI agents and the tools to build AI agents. Love connecting AI with other fields and yapping about all of them.
The jump from 10 to 13% fixes in 2025 to 46 to 47% in October would be easier to read if each fix pointed at the PR it repairs. git blame on the lines a fix touches gets you most of the way, and it would separate unread agent PRs from old code. I'd start with the 121 that mention Sentry
Ran it. For each October fix I took the lines it changed or deleted, blamed them at the commit just before the fix, and counted the PR that last touched most of them. 235 of the 278 fix PRs had lines I could trace to an earlier PR. Nearly all of the rest only added code.
192 of the 235, about 82%, mostly repaired lines from a PR whose description says Claude Code. That sounds bad until you put it next to the base rate: 84% of everything merged since April is an agent PR, so agent code isn't over-represented among the things that needed fixing. The worry it does settle is the one in the post about fixes for old code. Only 5 went back to code from before April. Blame only sees the last change to a line, so an agent refactor of old code counts as agent code, but that wouldn't hide much here.
The Sentry subset first needs a correction. Searching titles and full descriptions, I get 93 October PRs that mention Sentry, not 121, and I can't reproduce the original count. 52 of those are fixes, and 43 had lines to blame. 40 of the 43 repaired agent PRs, none touched pre-April code, and the median gap between the broken PR and its fix was 14 days, against 41 for fixes overall. So Sentry catches the agent's mistakes faster than everything else does, which is about what you'd hope it does.
GMT strategy and custom B2B outreach lists on request: people and companies by ICP, domain, profile URL, email or name. 1B+ profiles. Claude Code plugins, API, MCP. kevin@outricher.com
same habit with data. i never read the whole file but i always open five rows by hand before anything goes into a list, it catches more than any summary the tool gives me
We're a place where coders share, stay up-to-date and grow their careers.
Your last bullet, screenshots by default, is a rule in our release process at work. Every feature attaches two short videos before it reaches QA: one showing the product, one walking through the interfaces and the contract. The product owner and the dev lead check those first, then QA tests from the feature doc, the way a client would hit it, not from the diff. Did October's fixes cluster in the screens, or in the parts you trust the tests for?
Mostly in the parts I trust the tests for. Counting October PR titles that start with "fix", about 45% of the API's merged PRs were fixes and about 47% of the frontend's, so neither repo stands out, and the API had nearly twice as many in total. Of the frontend fixes, at most about a quarter look like screen problems by their titles: layout, scrolling, wrapping, mobile.
Almost every fix in both repos also touched a test file. I read that as the agent adding a regression test along with the fix, not the tests catching the bug beforehand. This is a rough count from titles, not a look at each PR.
Your two videos are close to what I said I should have done with screenshots from the start, just with the contract walkthrough added.
A regression test that lands with the fix records where the bug was, not who caught it. That is why our QA writes its cases from the feature doc at hand-off, while the dev builds, not after. How many of the October fixes started as a report from a user rather than your own review?
Almost none, going by the issues. Most of October's 278 fixes point back at an issue, 209 different ones. By the issue text, 20 started from Sentry, half of them through a button on our Discord alerts that files the issue straight from the error. 8 were bugs I reported myself, and 1 came from a player. The other 180 or so read like the agent's own findings from reading the code, and a lot of them are follow-ups to an earlier fix. I've only read a sample of those, so take that split as rough.
Which supports your point. The regression test says where the bug was. It took the issue history to say who found it, and by that count it was mostly the agent, not users and not me.
180 of 278 from the agent's own reading is more than I would have guessed. The follow-ups are the part I would count next. My experience is that agents are not deterministic: one run follows every rule, the next does not, so a fix can need a second fix. How soon after the first fix did the follow-ups usually land?
Quickly, for most of them. I checked it the git blame way: for each October fix, which earlier PR last wrote the lines it changed. For 82 of 236 fixes, that earlier PR was itself a fix. Half of those follow-ups landed within about 4 days of the first one. 23 came within a day and 50 within a week. The rest are spread out, some months later, and those are probably not a second try at the same bug.
Blame only shows who last touched the lines, not that it was the same bug, so it's a rough count in both directions. But it does fit what you describe. When a fix needs another fix, it mostly shows up in the same week.
Visibility is a good compression strategy, but the ranking signal needs a falsifiable link to risk. Diff size alone is noisy; boundary crossings, permission changes, generated-file churn, and behavior not covered by tests are often better review triggers. I’d preserve the reason a PR was surfaced alongside the PR, then audit false negatives after incidents to tune the queue.
The line that stuck with me is "read the description like it's the contract". I agree, and it's also the one input on your list that nothing checks against the diff. CI checks the code, you check the description against what you asked for, but the link between description and diff is taken on trust.
We had exactly that case. While we were testing a GitHub Action in our own test app, Claude Code made a commit titled "Limit contact name to 250 chars". The sed it ran hit the email rule instead, because the name was already max:200. Tests green, message perfectly sensible, wrong line changed. Since then we run a check that compares what the message claims with what the diff touches. On 937 commits from five PHP repos it flagged 23, about 2.5%, so it's quiet enough to leave on.
With descriptions at 2,400+ characters there's even more room for the text to drift from the code. Have you caught one of your 1,396 where the description was right about the intent and wrong about what actually changed?
Not that I can show you, because I haven't run anything like your check. I tried the cheap version on October's PRs, matching the scope in the title against the paths the diff touches, and it's useless here. 107 of 520 scoped titles didn't match any path, and the ones I went through were all feature names rather than folders: "grid rooms", "tenagoal", "guesses". The scope is how I think about the product, not where the code lives.
So a check like yours would have to read the claim and the diff, the way yours does, not compare two strings. Your sed example is exactly the case my setup wouldn't catch: tests green, description sensible, wrong line. 2.5% flagged on 937 commits is quiet enough to be worth trying.
Feature names instead of folders is the part that kills any string comparison, and it's the same reason a path check couldn't have caught the sed case: the title said the right thing about the right feature. The line that was wrong was simply not the one the sentence was about.
That's why our check starts from the claim, not from the scope. It takes what the commit message says it changed and looks for the changed lines that sentence cannot account for. On the 937 commits that gave 23 flags, and the ones I read were all message and diff disagreeing, not naming conventions.
If you try it on October's 520, the useful number is not the flag rate but how many flags you'd have wanted to see. I'd treat anything above a handful a week as a bug in the check, not in your PRs.
Interesting perspective! The increase in fix-related PRs is particularly telling. As AI increases development speed, I think QA becomes even more important. Green CI is valuable, but we also need confidence that we're testing the right things, not just that the tests pass.
1,241 agent PRs and still opening the ones you can actually look at — that's the honest version of review, not the pretend one. I do the same with screenshots and pages, and wave the rest through on green CI with a straight face. Did anything in the ones you did look at ever make you go back and re-check a merged one you hadn't?
The history can't say what prompted a second look, but it does show how often a merged PR got revisited. I blamed the lines each October fix changed back to the PR that last touched them. 81 of 235 fixes went back to something merged in the previous seven days, and 82 repaired a PR that was itself a fix.
So going back happens a lot. It just happens as another PR instead of a revert, which is why the revert count since April is still zero. Whether a preview or a Sentry alert started it is not something I can pull apart from the data. For the fixes that mention Sentry, the median gap from the broken PR to its fix was 14 days, against 41 for fixes overall, so the alerts at least get there sooner.
The jump from 10 to 13% fixes in 2025 to 46 to 47% in October would be easier to read if each fix pointed at the PR it repairs. git blame on the lines a fix touches gets you most of the way, and it would separate unread agent PRs from old code. I'd start with the 121 that mention Sentry
Ran it. For each October fix I took the lines it changed or deleted, blamed them at the commit just before the fix, and counted the PR that last touched most of them. 235 of the 278 fix PRs had lines I could trace to an earlier PR. Nearly all of the rest only added code.
192 of the 235, about 82%, mostly repaired lines from a PR whose description says Claude Code. That sounds bad until you put it next to the base rate: 84% of everything merged since April is an agent PR, so agent code isn't over-represented among the things that needed fixing. The worry it does settle is the one in the post about fixes for old code. Only 5 went back to code from before April. Blame only sees the last change to a line, so an agent refactor of old code counts as agent code, but that wouldn't hide much here.
The Sentry subset first needs a correction. Searching titles and full descriptions, I get 93 October PRs that mention Sentry, not 121, and I can't reproduce the original count. 52 of those are fixes, and 43 had lines to blame. 40 of the 43 repaired agent PRs, none touched pre-April code, and the median gap between the broken PR and its fix was 14 days, against 41 for fixes overall. So Sentry catches the agent's mistakes faster than everything else does, which is about what you'd hope it does.
same habit with data. i never read the whole file but i always open five rows by hand before anything goes into a list, it catches more than any summary the tool gives me