A 75% improvement is compelling, but I would publish the p95 and the cost of the loop alongside the headline number. Automated optimization can trade client latency for build time, regressions, or infra spend; a small before-and-after budget for those dimensions makes the result easier to reproduce and trust.
I'd ballpark additional spend on this at ~$180/mo on top of subscriptions we already were paying for.
The openhands instance and all its computer is about $100; our GitHub Actions spend is free under GitHub Open Source policy; and this is all re-using subscription credits we pay for otherwise. Then I'll add an extra $80 for certain misc spend along the way.
There's an inherent efficiency in that we use subsidized subscription tokens instead of API tokens. that benefit may eventually go away.
There are definitely people doing this sort of thing incredibly inefficiently, but not us. We're definitely frugal with this stuff. π
That breakdown makes the trade-off concrete. The subscription-credit advantage is useful while it exists, but a durable report should separate marginal cash cost from consumed included capacity and show the fallback price if that subsidy changes. Otherwise the system can look cheaper than it is precisely when it needs to scale.
Wow just wow, that's Engineering with a big E - for sure you guys always did great work, but this sounds like "next level" ...
Never heard of OpenHands, and I'm sure I'm nowhere near the "AI maturity level" which would make it make sense for me, but I've bookmarked it 'for the future'!
In a way I'd say this is the project that made me and our team get to "next level" in the first place. I'd definitely give it a try. There's also several products in the OpenHands spaceβ I really liked that this one is open source and self-hostable.
This way of doing software dev and managing the SDLC definitely sounds like "the future" of development - if you have a good grasp of how to do this (which you pretty clearly achieved with this project) then you're in a good place, also in the medium/long term ...
OpenHands looks interesting, although I'm far from an expert, or even moderately knowledgeable in this space - just getting my feet wet and taking baby steps with Claude Code, but I can imagine that the openness and extensibility give OpenHand the edge (especially for bigger/more complex projects) over a more "closed" environment like Claude Code ...
But for me OpenHands is probably something to try out in the future, when I'm a bit further along this path ...
I research AI tools, APIs & agents β comparing specs, pricing and real user reports β then publish honest rankings so you don't waste money. Research-first, never sponsored fluff.
70ms to 16ms, and P99 over a second down to something sane β that's the kind of boring work nobody prioritizes until it hurts. The 'API endpoint with a pulse' line is painfully accurate though, I spent weeks in exactly that spot, clipboard between dashboards and terminal. Genuine question: of the 475 PRs, how many were the loop finding real issues vs just churn? The garden metaphor only works if the soil holds.
I'd say 99% real issues, but the loop consistently hit local maximas where the improvements started being really small, so that forces tweaks to the system to re-evaluate the instructions on how to pursure finding the issues.
It could be possible to do higher level instructions so this didn't have to be tweaked, but as a matter of safety I think it's better to keep the loops highly instructed.
Hmmm could be an issue with our image CDN. Could you open an issue on github.com/forem/forem with all possible details you're seeing from console (this likely isn't a fix in our code, but maybe configβ but still best place to put a bug)
Python and AI engineer. Production systems for publishers, public institutions and companies: document and OCR pipelines, agents, verification. Founder of Devaland: Deal OS, DroneWatch AI, PawnMeter.
The line that stuck with me is the deterministic Python gates in front of every cron, so a full queue exits in two seconds with zero tokens. I learned the same thing the expensive way: the cheapest agent turn is the one a ten-line script decides not to start.
One question about the second-lens auditor. When a perf PR claims a gain, how does it tell a real improvement from a change in what got sampled? The worst bug I shipped this year was a guard that reported "all clean" because it was reading a format I had invented, so it could never see the failure it was meant to catch. Since then, every new check of mine has to prove it fails on a planted defect before I trust a green result.
The reason this works where most autonomous agent setups stall is the shape of the verification function. When the success criterion is subjective or requires reading code for subtle intent, human review time scales linearly with PR volume, which turns the engineer into a bottleneck. By anchoring the loop to live telemetry and automated test suites, the evaluation becomes an objective numeric check. That drives marginal verification cost low enough that merging 350 autonomous PRs a month actually frees up human attention instead of consuming it.
A 75% improvement is compelling, but I would publish the p95 and the cost of the loop alongside the headline number. Automated optimization can trade client latency for build time, regressions, or infrastructure spend; a small before-and-after budget for those dimensions makes the result easier to reproduce and trust.
The isolated worktrees and deterministic pre-flight checks seem like a useful way to prevent parallel runs from colliding or wasting cycles. How do you decide which changes qualify for the low-risk auto-merge path?
Founder of Hex16, a small web studio in Poland. I build Next.js sites and headless Shopify stores, with GSAP, Three.js and shaders where they earn their place, and look after technical SEO.
The part I'd steal is the first perf loop failing because it read the code and guessed. Humans do the same on client sites: we optimize whatever looks slow in the code, and the real cost turns out to be a 3 MB photo the client uploaded from their phone. Telemetry first, opinions second.
Did the loops ever ship something that improved the server numbers but made the page feel worse, like caching that served a stale feed?
The detail that stuck with me is the second-lens auditor on a different model family before auto-merge. That is the part most autonomous setups skip, and it explains how 475 PRs can land without the codebase quietly rotting. I also appreciated the honesty about the first perf loop failing because it read the code and guessed at bottlenecks instead of looking at live telemetry β that failure mode feels very familiar. Did the cross-family router end up saving money overall, or did the extra model calls roughly cancel out the subscription-credit advantage you mentioned?
The layer-by-layer breakdown is the most useful part here β DB 39β12ms, view 12β2.6ms, Redis 7β1.3ms β because that attribution is what lets an automated loop target the real bottleneck instead of guessing. On the desktop side we hit the same lesson: a feedback loop only stays trustworthy if each iteration records its own metrics and diff, so a regression in one layer can be rolled back without touching the others. That human-middleware warning maps well to Electron startup profiling too β babysitting flaky runs burns the same hours that would otherwise go to fixing the actual renderer/main-process hotspot.
The first perf loop failing because it read the code and guessed is the most useful detail in here. People do exactly the same thing, and the fix is the same too: start from what the telemetry says is slow. Once the loop was anchored to Better Stack numbers and the test suite, the check became a measurement rather than an opinion, and that is what let 351 PRs merge without anyone reading every line.
The answer about local maxima was honest as well. When the easy SQL wins dry up, the gains get smaller and the instructions need rework. Keeping the loops narrowly instructed for safety, rather than handing them a vague goal, seems like the right call at that volume.
At Ascendra Ventures we scope performance work the same way: agree the metric, the baseline, and the target before touching code, so it is clear when the job is done. Your layer breakdown, database 39 ms to under 12 and views 12 to 2.6, is exactly the kind of baseline that makes that possible.
How do you decide which PR types qualify for the auto-merge path, and has that list changed since October?
The telemetry first switch makes sense. For the L1 Redis cache, Iβd keep a cold-cache trace beside the warm one and replay the same request mix after a deploy. That separates a faster hit path from a worse miss path. One little cache exercise Iβd borrow for the regression suite is rewriting a key just before its old deadline, then checking that cleanup leaves the new value alone. Did any of the per-endpoint gains look different with a cold cache?
Programming in plain words β Python guides, free tools, and honest learning paths for beginners. Building AheadKit: programming books and practical digital tools.
The 70ms to 16ms arc is wild, but the bit I keep thinking about is the rust-loop being gated to one open PR so the precompiled binaries never conflict. Did that gate ever become the bottleneck once the easy SQL wins cleared, or did the 10-minute tick cadence keep it fed just slowly enough? Also "human middleware" and "an API endpoint with a pulse" might be the most accurate job description of agentic development I've read this year.
This is a compelling example of where agentic development becomes genuinely transformative: not simply generating code faster, but creating a system that continuously identifies, prioritizes, validates, and delivers improvements with minimal human intervention.
What stands out most is the importance of grounding autonomous agents in real production telemetry and enforcing strong boundaries around their work. Without measurable bottlenecks, isolated workspaces, deterministic checks, and independent review, an automated loop can just as easily multiply technical debt as reduce it. The engineering discipline behind the factory is arguably more impressive than the number of PRs it produces.
I also appreciate the distinction between systematic maintenance and bespoke engineering. Automating repetitive optimization work creates something more valuable than saved keystrokes: it gives developers the time and attention to tackle architectural decisions, user experience, and problems that require human judgment.
One important challenge as these systems scale will be preserving accountability. Every autonomous change still needs clear ownership, traceability, meaningful rollback paths, and safeguards against optimizations that improve one metric while quietly harming another.
The real promise of AI-assisted engineering isn't replacing the developer with an agent; it's building reliable systems that free developers to focus on the work that actually needs them.
The obsession with wrapping remote APIs into multi-layered agent abstractions is creating massive hidden latency and failure cascades in production.
If you profile real-world workloads, bare-metal local inference (via direct C++ / llama.cpp or vLLM bindings) consistently beats fragile cloud wrappers on both predictability and cost. Tooling should eliminate abstraction penalties, not compound them.
The decision to gate the Rust rewrite loop to a single open PR stood out as the clearest trade-off in the piece. With 32 native rewrites verified by 1,000-input differential tests, that gate avoids conflicting precompiled binaries, but it also looks like the main throughput bottleneck once the easy SQL wins are gone. The database drop from 39ms to under 12ms per request explains much of the 70ms to 16ms improvement, so I wonder how the team decides when a hot Ruby path is worth the slower, serialized Rust conversion rather than another round of query or caching work. Is there a latency or call-frequency threshold for promoting a path into that loop, or is it driven mainly by the profiling data from live telemetry?
I think the hardest part isn't getting agents to open PRs. It's making sure those PRs are actually useful and don't create more work for the team. That's what I found most interesting here.
We're a place where coders share, stay up-to-date and grow their careers.
A 75% improvement is compelling, but I would publish the p95 and the cost of the loop alongside the headline number. Automated optimization can trade client latency for build time, regressions, or infra spend; a small before-and-after budget for those dimensions makes the result easier to reproduce and trust.
I'd ballpark additional spend on this at ~$180/mo on top of subscriptions we already were paying for.
The openhands instance and all its computer is about $100; our GitHub Actions spend is free under GitHub Open Source policy; and this is all re-using subscription credits we pay for otherwise. Then I'll add an extra $80 for certain misc spend along the way.
There's an inherent efficiency in that we use subsidized subscription tokens instead of API tokens. that benefit may eventually go away.
There are definitely people doing this sort of thing incredibly inefficiently, but not us. We're definitely frugal with this stuff. π
That breakdown makes the trade-off concrete. The subscription-credit advantage is useful while it exists, but a durable report should separate marginal cash cost from consumed included capacity and show the fallback price if that subsidy changes. Otherwise the system can look cheaper than it is precisely when it needs to scale.
Wow just wow, that's Engineering with a big E - for sure you guys always did great work, but this sounds like "next level" ...
Never heard of OpenHands, and I'm sure I'm nowhere near the "AI maturity level" which would make it make sense for me, but I've bookmarked it 'for the future'!
In a way I'd say this is the project that made me and our team get to "next level" in the first place. I'd definitely give it a try. There's also several products in the OpenHands spaceβ I really liked that this one is open source and self-hostable.
This way of doing software dev and managing the SDLC definitely sounds like "the future" of development - if you have a good grasp of how to do this (which you pretty clearly achieved with this project) then you're in a good place, also in the medium/long term ...
OpenHands looks interesting, although I'm far from an expert, or even moderately knowledgeable in this space - just getting my feet wet and taking baby steps with Claude Code, but I can imagine that the openness and extensibility give OpenHand the edge (especially for bigger/more complex projects) over a more "closed" environment like Claude Code ...
But for me OpenHands is probably something to try out in the future, when I'm a bit further along this path ...
70ms to 16ms, and P99 over a second down to something sane β that's the kind of boring work nobody prioritizes until it hurts. The 'API endpoint with a pulse' line is painfully accurate though, I spent weeks in exactly that spot, clipboard between dashboards and terminal. Genuine question: of the 475 PRs, how many were the loop finding real issues vs just churn? The garden metaphor only works if the soil holds.
I'd say 99% real issues, but the loop consistently hit local maximas where the improvements started being really small, so that forces tweaks to the system to re-evaluate the instructions on how to pursure finding the issues.
It could be possible to do higher level instructions so this didn't have to be tweaked, but as a matter of safety I think it's better to keep the loops highly instructed.
i dong know if its only me or...
the loading never ends... the text ,... really fast...
my connection :
different browser still the same ....
^_^
Hmmm could be an issue with our image CDN. Could you open an issue on github.com/forem/forem with all possible details you're seeing from console (this likely isn't a fix in our code, but maybe configβ but still best place to put a bug)
Thanks Ben, I've opened the issue on GitHub with the network details and screenshots. Here is the link: github.com/forem/forem/issues/2473...
The line that stuck with me is the deterministic Python gates in front of every cron, so a full queue exits in two seconds with zero tokens. I learned the same thing the expensive way: the cheapest agent turn is the one a ten-line script decides not to start.
One question about the second-lens auditor. When a perf PR claims a gain, how does it tell a real improvement from a change in what got sampled? The worst bug I shipped this year was a guard that reported "all clean" because it was reading a format I had invented, so it could never see the failure it was meant to catch. Since then, every new check of mine has to prove it fails on a planted defect before I trust a green result.
The reason this works where most autonomous agent setups stall is the shape of the verification function. When the success criterion is subjective or requires reading code for subtle intent, human review time scales linearly with PR volume, which turns the engineer into a bottleneck. By anchoring the loop to live telemetry and automated test suites, the evaluation becomes an objective numeric check. That drives marginal verification cost low enough that merging 350 autonomous PRs a month actually frees up human attention instead of consuming it.
A 75% improvement is compelling, but I would publish the p95 and the cost of the loop alongside the headline number. Automated optimization can trade client latency for build time, regressions, or infrastructure spend; a small before-and-after budget for those dimensions makes the result easier to reproduce and trust.
The isolated worktrees and deterministic pre-flight checks seem like a useful way to prevent parallel runs from colliding or wasting cycles. How do you decide which changes qualify for the low-risk auto-merge path?
The part I'd steal is the first perf loop failing because it read the code and guessed. Humans do the same on client sites: we optimize whatever looks slow in the code, and the real cost turns out to be a 3 MB photo the client uploaded from their phone. Telemetry first, opinions second.
Did the loops ever ship something that improved the server numbers but made the page feel worse, like caching that served a stale feed?
The detail that stuck with me is the second-lens auditor on a different model family before auto-merge. That is the part most autonomous setups skip, and it explains how 475 PRs can land without the codebase quietly rotting. I also appreciated the honesty about the first perf loop failing because it read the code and guessed at bottlenecks instead of looking at live telemetry β that failure mode feels very familiar. Did the cross-family router end up saving money overall, or did the extra model calls roughly cancel out the subscription-credit advantage you mentioned?
The layer-by-layer breakdown is the most useful part here β DB 39β12ms, view 12β2.6ms, Redis 7β1.3ms β because that attribution is what lets an automated loop target the real bottleneck instead of guessing. On the desktop side we hit the same lesson: a feedback loop only stays trustworthy if each iteration records its own metrics and diff, so a regression in one layer can be rolled back without touching the others. That human-middleware warning maps well to Electron startup profiling too β babysitting flaky runs burns the same hours that would otherwise go to fixing the actual renderer/main-process hotspot.
The first perf loop failing because it read the code and guessed is the most useful detail in here. People do exactly the same thing, and the fix is the same too: start from what the telemetry says is slow. Once the loop was anchored to Better Stack numbers and the test suite, the check became a measurement rather than an opinion, and that is what let 351 PRs merge without anyone reading every line.
The answer about local maxima was honest as well. When the easy SQL wins dry up, the gains get smaller and the instructions need rework. Keeping the loops narrowly instructed for safety, rather than handing them a vague goal, seems like the right call at that volume.
At Ascendra Ventures we scope performance work the same way: agree the metric, the baseline, and the target before touching code, so it is clear when the job is done. Your layer breakdown, database 39 ms to under 12 and views 12 to 2.6, is exactly the kind of baseline that makes that possible.
How do you decide which PR types qualify for the auto-merge path, and has that list changed since October?
The telemetry first switch makes sense. For the L1 Redis cache, Iβd keep a cold-cache trace beside the warm one and replay the same request mix after a deploy. That separates a faster hit path from a worse miss path. One little cache exercise Iβd borrow for the regression suite is rewriting a key just before its old deadline, then checking that cleanup leaves the new value alone. Did any of the per-endpoint gains look different with a cold cache?
The 70ms to 16ms arc is wild, but the bit I keep thinking about is the rust-loop being gated to one open PR so the precompiled binaries never conflict. Did that gate ever become the bottleneck once the easy SQL wins cleared, or did the 10-minute tick cadence keep it fed just slowly enough? Also "human middleware" and "an API endpoint with a pulse" might be the most accurate job description of agentic development I've read this year.
This is a compelling example of where agentic development becomes genuinely transformative: not simply generating code faster, but creating a system that continuously identifies, prioritizes, validates, and delivers improvements with minimal human intervention.
What stands out most is the importance of grounding autonomous agents in real production telemetry and enforcing strong boundaries around their work. Without measurable bottlenecks, isolated workspaces, deterministic checks, and independent review, an automated loop can just as easily multiply technical debt as reduce it. The engineering discipline behind the factory is arguably more impressive than the number of PRs it produces.
I also appreciate the distinction between systematic maintenance and bespoke engineering. Automating repetitive optimization work creates something more valuable than saved keystrokes: it gives developers the time and attention to tackle architectural decisions, user experience, and problems that require human judgment.
One important challenge as these systems scale will be preserving accountability. Every autonomous change still needs clear ownership, traceability, meaningful rollback paths, and safeguards against optimizations that improve one metric while quietly harming another.
The real promise of AI-assisted engineering isn't replacing the developer with an agent; it's building reliable systems that free developers to focus on the work that actually needs them.
The garden analogy is nice and that many PRs in a few weeks backing that up is awesome π
The obsession with wrapping remote APIs into multi-layered agent abstractions is creating massive hidden latency and failure cascades in production.
If you profile real-world workloads, bare-metal local inference (via direct C++ / llama.cpp or vLLM bindings) consistently beats fragile cloud wrappers on both predictability and cost. Tooling should eliminate abstraction penalties, not compound them.
The decision to gate the Rust rewrite loop to a single open PR stood out as the clearest trade-off in the piece. With 32 native rewrites verified by 1,000-input differential tests, that gate avoids conflicting precompiled binaries, but it also looks like the main throughput bottleneck once the easy SQL wins are gone. The database drop from 39ms to under 12ms per request explains much of the 70ms to 16ms improvement, so I wonder how the team decides when a hot Ruby path is worth the slower, serialized Rust conversion rather than another round of query or caching work. Is there a latency or call-frequency threshold for promoting a path into that loop, or is it driven mainly by the profiling data from live telemetry?
This is great work Dev team. Which loop PR gave the biggest single drop??
I'd have to hunt down exactly which it was. Definitely October 14th
I think the hardest part isn't getting agents to open PRs. It's making sure those PRs are actually useful and don't create more work for the team. That's what I found most interesting here.