xulingfeng, lovely to see the Stratagems preoccupation turn up in non-fiction. #32 had an AI confessing it kept turning an incomplete observation into a complete state. Here it’s an agent working faithfully from an incomplete PRD. 😄
One thing in your numbers is worth more than the headline. 36 defects from AI, with 21 valid, and 11 found manually, with 9 valid by subtraction, so roughly 42% invalid for the agents against 18% for the people. Small sample, but part of that gap is the undocumented context your testers were carrying in their heads.
So the invalid rate measures the organisation, not the AI, and it’s the first number you’ve had that counts decisions made without updating the document. The trap is that it vanishes once you fix the PRD process. I’d tag each invalid defect by reason, and the “spec was stale” bucket becomes a drift counter that keeps working afterwards.
Thank you for writing this one as carefully as you write the fiction. 🙏
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
Wow, this is such a sharp breakdown. You nailed the parallel between Stratagems #32 and this real-world case — both are about incomplete information being treated as full truth.
You’re absolutely right about the false positive rate. It’s not a score for the AI itself; it’s a mirror reflecting our undocumented tribal knowledge. Just to add context: these numbers are only from one single project. We’ve seen very different figures on another project, so we still need case data from more projects to draw solid conclusions.
Tagging invalid defects by root cause and tracking spec drift long-term is a great actionable suggestion. That’s exactly what we plan to build next.
Also, I just woke up from a nap. Those red notification numbers startled me, haha — looks like you’re quite the like spammer!
Thanks so much for reading closely and picking apart the numbers!🤣🤣🤣
Caught red-handed. I do leave a trail of hearty reactions on anything I like, and I’m not sorry about that part. Genuinely sorry about the nap, though. 🤣
But hold on. You’re the QA engineer here. You ran a system with no rate limiting on inbound notifications, in a known hostile environment, while unconscious. I switch everything off before a power nap. Why didn’t you?
Severity: Critical. Repro rate: 100%, every time I open your comments. Recommendation: block the release of my reaction finger immediately.
On the serious bit, about your comment, I’d push it one step further. If one project sits around 42% and another looks nothing like it, then the spread between projects is the measurement. A single project gives you a number. Several give you a map of which teams actually write things down.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
Haha, severity Critical accepted! This notification system definitely lacks proper rate limiting and snooze mode for napping hours. I’ll add this bug to my personal backlog 😂
You’re totally right — the variance across projects is the far more interesting metric than any single number. That spread really reveals the gap between teams on documentation habits. Yeah, agent-based automated testing still needs plenty of real-world project data. I plan to run more rounds of practical tests across different projects and put together a dedicated analysis post once I have enough data.
Thanks for being considerate, but I’m fully awake now hahahaha🤣
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
What a great article! I really need to think about implementing something similar in our project, because we're facing a very similar problem. Knowledge gets scattered, people leave, nobody updates the documentation, and important information just gets lost along the way.
Such interesting times ahead! Looking forward to more articles like this!
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
Thank you! I think this is exactly where AI could make a real difference.
But there's one thing I've learned from years in software testing: simply putting all the information into a knowledge base doesn't solve the problem. The real challenge is keeping that knowledge accurate and aligned with how the product actually works.
That's one of the things we're trying to explore: using AI not just to retrieve information, but also to help identify gaps, outdated assumptions, and inconsistencies between documentation and reality.
We actually have another interesting idea here. We're working on a hardware project — a voice recorder card. Imagine people carrying this little device around, recording conversations and important information throughout the day. AI could then automatically organize the recordings and sync the latest information into the knowledge base.
Sounds great in theory, right? I'm just a little worried the device might start making things up. Haha! 🤣
We're still figuring things out on our side, so I'll share more as the experiment evolves!
Of course, my next article might be Stratagems #33 instead. You never know! Haha! 😆
मैं आपकी इस बात से 100% सहमत हूँ। आपने वाकई बहुत अच्छी बात कही है! I don't believe AI will turn everyone into a programmer, nor do I think software teams will no longer need professional roles such as product, development, and QA.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
I love this perspective! AI is a powerful helper, yet specialized roles like QA, product and development carry critical context and judgement that no automated tool can fully take over.🙌
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
Love how an AI testing demo turned into the real conversation: once tests get cheap, the bottleneck moves to everything around them. The single test environment is a perfect example. Before adding a second static one, it might be worth looking at short-lived preview environments per branch or requirement, spun up on push and torn down on merge. Parallel work stops colliding, and your AI test agents each get a clean target instead of fighting over one. Looking forward to the next Stratagems chapter too!
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
That’s a fantastic suggestion! 👏Short-lived preview environments are exactly the direction we’re exploring next. A single shared test environment becomes a huge pain once many AI test agents run in parallel — they step on each other’s data constantly. Ephemeral environments per branch would give every agent an isolated clean state. And thanks, I’m hard at work on the next Stratagems chapter!😂
Automating test generation usually reveals that test authoring was never the scarce resource. When generation costs fall toward zero, the bottleneck shifts to specification stability and defect triage. If acceptance criteria drift across sprints, high-volume synthetic tests mostly produce a flood of false-positive regressions that human developers still have to inspect and dismiss. Teams end up rewriting how they define requirements because running ten thousand automated checks against ambiguous specs costs more in review time than writing fewer tests by hand.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
This is exactly one of the key observations in the post.
Test‑authoring is no longer the bottleneck once AI generates tests cheaply. What surfaces instead is messy, drifting acceptance criteria that human testers historically absorbed via ad‑hoc chats and tribal knowledge.
AI agents can’t pick up those off‑the‑record conversations. They strictly work from written specs. When specs fall behind real‑world decisions, we get floods of false‑positive regression findings.
That’s why our discussion shifted beyond testing tools: high‑volume synthetic test output forces the whole team to fix requirement alignment, otherwise the triage overhead eats away all efficiency gains. Really appreciate you calling this out!👍
Hi! Wouldn't it be amazing if we have one platform where we can install ai model locally and do everything we want in our daily lives there only. Like there are some apps but we have to connect them integrate many other apps why not just 1.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
That would be great! I'd definitely love to see something like that.
But I think the real challenge isn't just putting everything into one platform. It's making all those capabilities work reliably together.
Different tools have different data structures, permissions, workflows, and failure modes. Even if we put them behind one interface, those integration problems don't magically disappear.
Still, a well-designed local AI platform that handles all of this seamlessly would be incredibly useful!👍
That's exactly the problem I'm trying to solve with Oronode Lumina.
I'm building an all-in-one, local-first AI workspace that brings together the tools and capabilities people currently have to use across multiple separate applications.
The vision is simple: one platform where you can chat with AI, run local models, write and debug code, automate workflows, generate images and videos, work with files, connect plugins and MCP servers, and interact with your desktop — all within one unified environment.
Here's what I'm working to bring into Lumina:
Local AI inference: Run compatible GGUF models directly on your own hardware.
Multi-provider AI: Connect supported AI providers and local inference engines.
AI coding workspace: Coding assistance, project workflows, and integrated terminal tools.
Image and video generation: Creative AI capabilities in one workspace.
Audio intelligence: Speech processing, OCR, audio understanding, and video understanding.
Automation: Build and execute AI-powered workflows. It comes with a Goal mode which will keep on working in the background on a specific task while you step away doing productive things.
Plugins and MCP: Extend the platform with integrations and custom tools.
Computer control: Let AI interact with supported desktop workflows, with permission controls.
Remote access: Access supported desktop capabilities through a browser.
Analytics and personalisation: Customise the experience, interface, models, tools, and workflows.
Lumina also supports 3d visuals creation
Privacy and security are important parts of the architecture. Lumina is designed around local-first processing, secure credential handling, permission-based tool access, and user control over integrations. I'm working to make these protections part of the actual implementation, not just marketing promises. Most importantly all the data management in Lumina is End to End encrypted hence no server can ever read your data, messages or responses.
I'm aiming to launch the first release within this month.
I genuinely appreciate the interest and feedback. I'm building this to solve a problem I believe many people working with AI are experiencing: having too many disconnected tools and workflows.
More updates coming soon.
Oronode Lumina — One workspace. Your AI. Your control.
Great write-up, and I understand your point about how this process has ultimately lead to a rethinking of your company's development organization, and that's super cool ...
However, when I read what you accomplished in bullet point 1:
"1. 🤖 I Wanted AI to Handle the Entire Testing Process"
... from a technical/professional point of view, that's what I found the most impressive! ... even though I "get it" that you then go on to elaborate on the organizational "big picture" thing :-)
But yes, from a conceptual/philosophical point of view you're right - AI has the potential to "break down siloes" and reduce the friction in organizational processes, and I agree that ultimately that's the "big prize" which will really make companies realize the sought-after productivity improvements with AI ...
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
Thanks a lot for this thoughtful comment! Totally get that the idea of end‑to‑end AI testing is where it all started. Honestly, my very first goal was to let AI take care of the whole testing workflow. And yeah, let’s be real — part of me just wanted more breathing room. If AI could carry the load, I’d have time for other stuff, including goofing off a little 😂
But as it turned out, the biggest roadblocks weren’t just technical. They were team silos and messy, out‑of‑date requirements that nobody bothered to write down properly. That’s how we wound up rethinking things across the whole company, instead of just rolling out a new testing tool. Knocking down those silos is really where the real value lives.
Yep I totally get it - knocking down those walls and silos is the holy grail that companies are looking for, and which will ultimately unlock those productivity benefits through AI ! So yeah, that's the "big prize" ...
It's just that, as a 'techie', I couldn't help but being impressed by how you 'automated' that whole test cycle so effectively (bullet point 1) :-)
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
Haha, maybe deep down I’ve never fully trusted AI. After all, I’ve written so many posts about AI failures. That’s why I pushed the automated‑testing agents I built really hard right from the start — heavy prompting with all kinds of tough feedback language, plus rigorous review and iteration methods. If a human had to go through this level of grind, they’d probably want to pick a fight with me 😂😂😂
Our company helped a lot too; they gave us plenty of token quota to make this possible. There’s no way I could afford running this out of my own pocket.🤣
Interesting that a testing talk became a conversation about how the organization works. Generating more test cases won't resolve product, engineering, and QA having different definitions of “done.” What changed first afterward: ownership, acceptance criteria, or the review process?
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
That’s a really good question! Honestly none of these things can be fixed in isolation — they’re all tangled up. Different ideas of what “done” means bleed into acceptance criteria, ownership, and how reviews work. Just churning out more test cases will never fix the root issue, exactly like you said. We’re sorting everything out little by little, and we now have a dedicated weekly sync to keep track of all these efficiency improvements.
I research AI tools, APIs & agents — comparing specs, pricing and real user reports — then publish honest rankings so you don't waste money. Research-first, never sponsored fluff.
The stale-PRD bit is the one that hit me — the AI wasn't wrong, the doc was just quietly out of date, and manual testers had been papering over that gap with hallway conversations for years. That's the part I'd steal: the defect report is only as good as the shared understanding behind it. Did the founders' discussion actually change how PRDs get updated now, or is that still the part everyone agrees on in the meeting and forgets on Monday?
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
It can’t get brushed aside anymore. Our founder officially kicked off this initiative, and we have weekly syncs moving forward. Still, this is definitely a long game.
Requirements falling out of sync is usually where the testing problem really starts, so it makes sense the talk moved there. When the AI generates test cases from requirements, who signs off that a case matches what the product owner meant? That handoff looks like the new job.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
The stale-requirements finding is the part that will stay with me. AI flagged 21 of the 30 confirmed defects, but the ones marked invalid were not really AI mistakes — the official requirements had quietly drifted from what product and development had actually agreed in conversation. That reframes the false-positive rate as a measure of organizational drift, not model quality, which is a much more useful way to read those numbers.
I also liked the practical detail about defect reports: finding issues overnight is only half the job if the report is a dense block of text nobody can act on. The shift to readable reports, and to AI running overnight while humans review in the morning, feels like the right division of labor — the machine explores, the people decide.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
Such a great way to interpret those false positives! They aren’t just AI errors, they reveal gaps between documented requirements and what the team actually agreed on.
The overnight‑run + human morning review setup has become our sweet spot too. AI digs up potential issues, but clean, actionable reports are what make all that exploration useful. Yep, we’re braced for the long haul ahead.💪
Programming in plain words — Python guides, free tools, and honest learning paths for beginners. Building AheadKit: programming books and practical digital tools.
The part that stuck with me isn't the 70% — 21 of the 30 confirmed defects found by AI is impressive, but it's the stale-PRD story that actually matters. AI flagged a discrepancy because product and dev had quietly agreed on a change and never updated the document, and suddenly the "bug" was really an org problem wearing a test report as a disguise. That's the line I'd put on a slide: automation doesn't just test your code, it audits whether your team actually shares one version of the truth. Most teams don't, they just have testers patiently papering over it. The overnight loop is neat too, but the PRD finding is the one that changes how a company works.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
This is such a sharp way to put it! I love that line — automation audits whether the team shares one version of the truth. Exactly what we ran into. Testers used to fill those gaps silently, and AI just makes these hidden mismatches impossible to ignore.
Programming in plain words — Python guides, free tools, and honest learning paths for beginners. Building AheadKit: programming books and practical digital tools.
That silent part is what stuck with me too — testers absorbing the drift in hallway chats, and nobody noticing until the AI takes the written word literally. I ran into the same shape on my side last week: my agent followed a fake fix hiding in its own test output, because the real rule lived in someone's head and never made it into the doc. The audit framing is exactly right — it's less about catching bugs and more about finding out what your team never actually agreed on paper.
JavaScript enthusiast with a growing passion for Python. I love exploring the world of programming, experimenting with code, and discovering new ways to turn ideas into reality.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
JavaScript enthusiast with a growing passion for Python. I love exploring the world of programming, experimenting with code, and discovering new ways to turn ideas into reality.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
JavaScript enthusiast with a growing passion for Python. I love exploring the world of programming, experimenting with code, and discovering new ways to turn ideas into reality.
The overnight testing setup is interesting. One thing I'd watch is whether agents start reporting the same issue across repeated runs. Without some deduplication, the morning review could become its own full-time job. I've seen how quickly that gets messy.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
I actually thought about this when designing the automated testing workflow. We already have a deduplication mechanism in place.
Since the company currently gives us a practically unlimited token budget, agents check for existing duplicate bug reports before submitting a new one. 😆
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
the cheap gate on generated tests isn't review, it's a red run: git worktree add /tmp/pre
, point the new test at that tree, and require a non-zero exit before the
test is allowed anywhere near the branch. a generated test that passes on the broken build is
a characterization test -- it will pin the bug and read as coverage for six months until
someone deletes the assertion as redundant.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
Great insight! Validating generated tests against a broken build is a smart filter. Tests that blindly accept faulty behaviour only inflate coverage without real value, exactly the trap we watch out for. Worth experimenting with.
This is the part that would actually change how a small team works: the AI wasn't mostly wrong, the requirements had drifted. I've seen the same thing outside software — the agreement changes in a conversation, the written spec doesn't, and then someone gets blamed for following the old version. Making the doc update part of the decision, not a chore for later, feels like the real fix here. The overnight run plus a human morning review is also the right split — let the machine do the searching, and keep the judgment with people.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
So true! Specs falling behind verbal agreements is a cross‑industry pain point. Updating docs alongside decisions, not later, is the real fix. Machine searches, humans judge.
I write about Next.js + TypeScript + AI engineering, with a focus on making sites discoverable to LLMs and AI search. Long-form notes at mudassirkhan.me
the second review before a defect gets assigned is the piece that makes this usable, because a flood of unverified agent bug reports is how a dev team learns to ignore the tracker. in our agent testing the biggest time sink was false positives where the page was simply mid load, so we added a rule that a defect must reproduce on a fresh session before it files. locator maintenance dropping is real, but triage cost tends to replace it. how are you measuring the false positive rate on filed defects, and does it jump after a UI redesign?
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
We’ve run into lots of pitfalls here too. When I review the defects submitted by the AI, I’m also checking against that same outdated requirement document. That’s exactly what pushed us to keep specs updated as early and often as we can.
For our metrics, we count a defect as false positive once it gets sent back by engineers or marked as “not to fix” by product after review. Moving forward, we plan to have the AI scan for requirement changes before every run, update its test cases accordingly, and restart the testing cycle. At the end of the day, the core thing is keeping requirements complete and synced in a timely manner.
That AI testing session sounds just like our plant meetings at The Printing World when we started automating material tracking. Good tools always end up exposing messy handoffs!
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
I am a freelance Data Scientist with a proven record of delivering high‑impact, data‑driven solutions that strengthen strategy, security, and innovation.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
Hey, I'm Hema 👋
Developer, writer, and creator of Dev Opportunity Radar, a weekly series published every Friday on DEV, helping people discover opportunities they might otherwise miss.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
So true! My automated testing agent can handle lots of complex test scenarios, but it can’t fix people forgetting to update the specs after offline discussions. That human workflow piece is impossible to automate easily 😂
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
xulingfeng, lovely to see the Stratagems preoccupation turn up in non-fiction. #32 had an AI confessing it kept turning an incomplete observation into a complete state. Here it’s an agent working faithfully from an incomplete PRD. 😄
One thing in your numbers is worth more than the headline. 36 defects from AI, with 21 valid, and 11 found manually, with 9 valid by subtraction, so roughly 42% invalid for the agents against 18% for the people. Small sample, but part of that gap is the undocumented context your testers were carrying in their heads.
So the invalid rate measures the organisation, not the AI, and it’s the first number you’ve had that counts decisions made without updating the document. The trap is that it vanishes once you fix the PRD process. I’d tag each invalid defect by reason, and the “spec was stale” bucket becomes a drift counter that keeps working afterwards.
Thank you for writing this one as carefully as you write the fiction. 🙏
Wow, this is such a sharp breakdown. You nailed the parallel between Stratagems #32 and this real-world case — both are about incomplete information being treated as full truth.
You’re absolutely right about the false positive rate. It’s not a score for the AI itself; it’s a mirror reflecting our undocumented tribal knowledge. Just to add context: these numbers are only from one single project. We’ve seen very different figures on another project, so we still need case data from more projects to draw solid conclusions.
Tagging invalid defects by root cause and tracking spec drift long-term is a great actionable suggestion. That’s exactly what we plan to build next.
Also, I just woke up from a nap. Those red notification numbers startled me, haha — looks like you’re quite the like spammer!
Thanks so much for reading closely and picking apart the numbers!🤣🤣🤣
Caught red-handed. I do leave a trail of hearty reactions on anything I like, and I’m not sorry about that part. Genuinely sorry about the nap, though. 🤣
But hold on. You’re the QA engineer here. You ran a system with no rate limiting on inbound notifications, in a known hostile environment, while unconscious. I switch everything off before a power nap. Why didn’t you?
Severity: Critical. Repro rate: 100%, every time I open your comments. Recommendation: block the release of my reaction finger immediately.
On the serious bit, about your comment, I’d push it one step further. If one project sits around 42% and another looks nothing like it, then the spread between projects is the measurement. A single project gives you a number. Several give you a map of which teams actually write things down.
Now go back to sleep. I’ll try to be quiet. 🙏
Haha, severity Critical accepted! This notification system definitely lacks proper rate limiting and snooze mode for napping hours. I’ll add this bug to my personal backlog 😂
You’re totally right — the variance across projects is the far more interesting metric than any single number. That spread really reveals the gap between teams on documentation habits. Yeah, agent-based automated testing still needs plenty of real-world project data. I plan to run more rounds of practical tests across different projects and put together a dedicated analysis post once I have enough data.
Thanks for being considerate, but I’m fully awake now hahahaha🤣
🤣🤣
哈哈哈哈🤣强迫症看了受不了啊
What a great article! I really need to think about implementing something similar in our project, because we're facing a very similar problem. Knowledge gets scattered, people leave, nobody updates the documentation, and important information just gets lost along the way.
Such interesting times ahead! Looking forward to more articles like this!
Thank you! I think this is exactly where AI could make a real difference.
But there's one thing I've learned from years in software testing: simply putting all the information into a knowledge base doesn't solve the problem. The real challenge is keeping that knowledge accurate and aligned with how the product actually works.
That's one of the things we're trying to explore: using AI not just to retrieve information, but also to help identify gaps, outdated assumptions, and inconsistencies between documentation and reality.
We actually have another interesting idea here. We're working on a hardware project — a voice recorder card. Imagine people carrying this little device around, recording conversations and important information throughout the day. AI could then automatically organize the recordings and sync the latest information into the knowledge base.
Sounds great in theory, right? I'm just a little worried the device might start making things up. Haha! 🤣
We're still figuring things out on our side, so I'll share more as the experiment evolves!
Of course, my next article might be Stratagems #33 instead. You never know! Haha! 😆
मैं आपकी इस बात से 100% सहमत हूँ। आपने वाकई बहुत अच्छी बात कही है!
I don't believe AI will turn everyone into a programmer, nor do I think software teams will no longer need professional roles such as product, development, and QA.
I love this perspective! AI is a powerful helper, yet specialized roles like QA, product and development carry critical context and judgement that no automated tool can fully take over.🙌
But that was your point, right? I agree with what you said.
👍🤣👍Thanks for the approval from this top student, haha 😆
Love this line. Using AI to extend our capabilities rather than replace humans entirely.
Thanks! AI is our force multiplier, not a replacement. It runs overnight, and we steer the decisions.😆
😂
🤣
🤣
Love how an AI testing demo turned into the real conversation: once tests get cheap, the bottleneck moves to everything around them. The single test environment is a perfect example. Before adding a second static one, it might be worth looking at short-lived preview environments per branch or requirement, spun up on push and torn down on merge. Parallel work stops colliding, and your AI test agents each get a clean target instead of fighting over one. Looking forward to the next Stratagems chapter too!
That’s a fantastic suggestion! 👏Short-lived preview environments are exactly the direction we’re exploring next. A single shared test environment becomes a huge pain once many AI test agents run in parallel — they step on each other’s data constantly. Ephemeral environments per branch would give every agent an isolated clean state. And thanks, I’m hard at work on the next Stratagems chapter!😂
Automating test generation usually reveals that test authoring was never the scarce resource. When generation costs fall toward zero, the bottleneck shifts to specification stability and defect triage. If acceptance criteria drift across sprints, high-volume synthetic tests mostly produce a flood of false-positive regressions that human developers still have to inspect and dismiss. Teams end up rewriting how they define requirements because running ten thousand automated checks against ambiguous specs costs more in review time than writing fewer tests by hand.
This is exactly one of the key observations in the post.
Test‑authoring is no longer the bottleneck once AI generates tests cheaply. What surfaces instead is messy, drifting acceptance criteria that human testers historically absorbed via ad‑hoc chats and tribal knowledge.
AI agents can’t pick up those off‑the‑record conversations. They strictly work from written specs. When specs fall behind real‑world decisions, we get floods of false‑positive regression findings.
That’s why our discussion shifted beyond testing tools: high‑volume synthetic test output forces the whole team to fix requirement alignment, otherwise the triage overhead eats away all efficiency gains. Really appreciate you calling this out!👍
Hi! Wouldn't it be amazing if we have one platform where we can install ai model locally and do everything we want in our daily lives there only. Like there are some apps but we have to connect them integrate many other apps why not just 1.
That would be great! I'd definitely love to see something like that.
But I think the real challenge isn't just putting everything into one platform. It's making all those capabilities work reliably together.
Different tools have different data structures, permissions, workflows, and failure modes. Even if we put them behind one interface, those integration problems don't magically disappear.
Still, a well-designed local AI platform that handles all of this seamlessly would be incredibly useful!👍
That's exactly the problem I'm trying to solve with Oronode Lumina.
I'm building an all-in-one, local-first AI workspace that brings together the tools and capabilities people currently have to use across multiple separate applications.
The vision is simple: one platform where you can chat with AI, run local models, write and debug code, automate workflows, generate images and videos, work with files, connect plugins and MCP servers, and interact with your desktop — all within one unified environment.
Here's what I'm working to bring into Lumina:
Privacy and security are important parts of the architecture. Lumina is designed around local-first processing, secure credential handling, permission-based tool access, and user control over integrations. I'm working to make these protections part of the actual implementation, not just marketing promises. Most importantly all the data management in Lumina is End to End encrypted hence no server can ever read your data, messages or responses.
I'm aiming to launch the first release within this month.
I genuinely appreciate the interest and feedback. I'm building this to solve a problem I believe many people working with AI are experiencing: having too many disconnected tools and workflows.
More updates coming soon.
Oronode Lumina — One workspace. Your AI. Your control.
Great write-up, and I understand your point about how this process has ultimately lead to a rethinking of your company's development organization, and that's super cool ...
However, when I read what you accomplished in bullet point 1:
"1. 🤖 I Wanted AI to Handle the Entire Testing Process"
... from a technical/professional point of view, that's what I found the most impressive! ... even though I "get it" that you then go on to elaborate on the organizational "big picture" thing :-)
But yes, from a conceptual/philosophical point of view you're right - AI has the potential to "break down siloes" and reduce the friction in organizational processes, and I agree that ultimately that's the "big prize" which will really make companies realize the sought-after productivity improvements with AI ...
Thanks a lot for this thoughtful comment! Totally get that the idea of end‑to‑end AI testing is where it all started. Honestly, my very first goal was to let AI take care of the whole testing workflow. And yeah, let’s be real — part of me just wanted more breathing room. If AI could carry the load, I’d have time for other stuff, including goofing off a little 😂
But as it turned out, the biggest roadblocks weren’t just technical. They were team silos and messy, out‑of‑date requirements that nobody bothered to write down properly. That’s how we wound up rethinking things across the whole company, instead of just rolling out a new testing tool. Knocking down those silos is really where the real value lives.
Yep I totally get it - knocking down those walls and silos is the holy grail that companies are looking for, and which will ultimately unlock those productivity benefits through AI ! So yeah, that's the "big prize" ...
It's just that, as a 'techie', I couldn't help but being impressed by how you 'automated' that whole test cycle so effectively (bullet point 1) :-)
Haha, maybe deep down I’ve never fully trusted AI. After all, I’ve written so many posts about AI failures. That’s why I pushed the automated‑testing agents I built really hard right from the start — heavy prompting with all kinds of tough feedback language, plus rigorous review and iteration methods. If a human had to go through this level of grind, they’d probably want to pick a fight with me 😂😂😂
Our company helped a lot too; they gave us plenty of token quota to make this possible. There’s no way I could afford running this out of my own pocket.🤣
Interesting that a testing talk became a conversation about how the organization works. Generating more test cases won't resolve product, engineering, and QA having different definitions of “done.” What changed first afterward: ownership, acceptance criteria, or the review process?
That’s a really good question! Honestly none of these things can be fixed in isolation — they’re all tangled up. Different ideas of what “done” means bleed into acceptance criteria, ownership, and how reviews work. Just churning out more test cases will never fix the root issue, exactly like you said. We’re sorting everything out little by little, and we now have a dedicated weekly sync to keep track of all these efficiency improvements.
The stale-PRD bit is the one that hit me — the AI wasn't wrong, the doc was just quietly out of date, and manual testers had been papering over that gap with hallway conversations for years. That's the part I'd steal: the defect report is only as good as the shared understanding behind it. Did the founders' discussion actually change how PRDs get updated now, or is that still the part everyone agrees on in the meeting and forgets on Monday?
It can’t get brushed aside anymore. Our founder officially kicked off this initiative, and we have weekly syncs moving forward. Still, this is definitely a long game.
Requirements falling out of sync is usually where the testing problem really starts, so it makes sense the talk moved there. When the AI generates test cases from requirements, who signs off that a case matches what the product owner meant? That handoff looks like the new job.
Yep, we’re braced for the long haul ahead.😂
The stale-requirements finding is the part that will stay with me. AI flagged 21 of the 30 confirmed defects, but the ones marked invalid were not really AI mistakes — the official requirements had quietly drifted from what product and development had actually agreed in conversation. That reframes the false-positive rate as a measure of organizational drift, not model quality, which is a much more useful way to read those numbers.
I also liked the practical detail about defect reports: finding issues overnight is only half the job if the report is a dense block of text nobody can act on. The shift to readable reports, and to AI running overnight while humans review in the morning, feels like the right division of labor — the machine explores, the people decide.
Such a great way to interpret those false positives! They aren’t just AI errors, they reveal gaps between documented requirements and what the team actually agreed on.
The overnight‑run + human morning review setup has become our sweet spot too. AI digs up potential issues, but clean, actionable reports are what make all that exploration useful. Yep, we’re braced for the long haul ahead.💪
The part that stuck with me isn't the 70% — 21 of the 30 confirmed defects found by AI is impressive, but it's the stale-PRD story that actually matters. AI flagged a discrepancy because product and dev had quietly agreed on a change and never updated the document, and suddenly the "bug" was really an org problem wearing a test report as a disguise. That's the line I'd put on a slide: automation doesn't just test your code, it audits whether your team actually shares one version of the truth. Most teams don't, they just have testers patiently papering over it. The overnight loop is neat too, but the PRD finding is the one that changes how a company works.
This is such a sharp way to put it! I love that line — automation audits whether the team shares one version of the truth. Exactly what we ran into. Testers used to fill those gaps silently, and AI just makes these hidden mismatches impossible to ignore.
That silent part is what stuck with me too — testers absorbing the drift in hallway chats, and nobody noticing until the AI takes the written word literally. I ran into the same shape on my side last week: my agent followed a fake fix hiding in its own test output, because the real rule lived in someone's head and never made it into the doc. The audit framing is exactly right — it's less about catching bugs and more about finding out what your team never actually agreed on paper.
🥰
😁
🗽
🤣👍
Thanks🗽🗽🗽🗽
The overnight testing setup is interesting. One thing I'd watch is whether agents start reporting the same issue across repeated runs. Without some deduplication, the morning review could become its own full-time job. I've seen how quickly that gets messy.
I actually thought about this when designing the automated testing workflow. We already have a deduplication mechanism in place.
Since the company currently gives us a practically unlimited token budget, agents check for existing duplicate bug reports before submitting a new one. 😆
the cheap gate on generated tests isn't review, it's a red run: git worktree add /tmp/pre
, point the new test at that tree, and require a non-zero exit before the
test is allowed anywhere near the branch. a generated test that passes on the broken build is
a characterization test -- it will pin the bug and read as coverage for six months until
someone deletes the assertion as redundant.
Great insight! Validating generated tests against a broken build is a smart filter. Tests that blindly accept faulty behaviour only inflate coverage without real value, exactly the trap we watch out for. Worth experimenting with.
This is the part that would actually change how a small team works: the AI wasn't mostly wrong, the requirements had drifted. I've seen the same thing outside software — the agreement changes in a conversation, the written spec doesn't, and then someone gets blamed for following the old version. Making the doc update part of the decision, not a chore for later, feels like the real fix here. The overnight run plus a human morning review is also the right split — let the machine do the searching, and keep the judgment with people.
So true! Specs falling behind verbal agreements is a cross‑industry pain point. Updating docs alongside decisions, not later, is the real fix. Machine searches, humans judge.
the second review before a defect gets assigned is the piece that makes this usable, because a flood of unverified agent bug reports is how a dev team learns to ignore the tracker. in our agent testing the biggest time sink was false positives where the page was simply mid load, so we added a rule that a defect must reproduce on a fresh session before it files. locator maintenance dropping is real, but triage cost tends to replace it. how are you measuring the false positive rate on filed defects, and does it jump after a UI redesign?
We’ve run into lots of pitfalls here too. When I review the defects submitted by the AI, I’m also checking against that same outdated requirement document. That’s exactly what pushed us to keep specs updated as early and often as we can.
For our metrics, we count a defect as false positive once it gets sent back by engineers or marked as “not to fix” by product after review. Moving forward, we plan to have the AI scan for requirement changes before every run, update its test cases accordingly, and restart the testing cycle. At the end of the day, the core thing is keeping requirements complete and synced in a timely manner.
That AI testing session sounds just like our plant meetings at The Printing World when we started automating material tracking. Good tools always end up exposing messy handoffs!
Exactly, great tools reveal the human‑process pain points we used to overlook.
I agree with you boss! I think that DSA and System Design is still important in the age of AI
🤣Thank you
Honestly, getting everyone on the same page with specs is way harder than building the actual tools. That workflow side of things is the real test 😄
So true! My automated testing agent can handle lots of complex test scenarios, but it can’t fix people forgetting to update the specs after offline discussions. That human workflow piece is impossible to automate easily 😂
That’s interesting! This post has stayed on the front page for so many days. My previous articles only lasted a day at most on the homepage😂