Developer educator and advocate who still ships code. 15 years teaching developers at MongoDB, Heroku, Cisco, and Treehouse. Now writing about AI-assisted development, agent memory, and provenance.
Oh, this is actually a really good point! Maybe this is the missing piece we've been looking for when it comes to programming with AI agents. Of course, we're all basically trying to predict the future here, so take this with a grain of salt. :D
But think about airline pilots. Modern aircraft spend most of their flight time on autopilot. The plane can practically fly itself, and there are countless safety systems in place. And yet, I suspect most people would still be pretty uncomfortable boarding a fully autonomous passenger plane with no human pilots on board.
And here's the interesting part: pilots spend a lot of time training in simulators, learning how to handle emergencies that they might never encounter during an actual flight. They're not just trained to operate the aircraft under normal conditions. They're trained to understand what's happening and take control when something goes terribly wrong.
What if software engineering eventually moves in a similar direction?
Maybe future developers won't need to spend years writing routine code to become experienced engineers. Maybe they'll learn through simulations, deliberately broken systems, security incidents, debugging exercises, and scenarios where AI agents fail in unexpected ways. Instead of learning primarily by writing code, they'll learn by understanding, questioning, and fixing it.
And perhaps that could even help solve the junior developer problem we've been discussing. If the traditional path from junior to senior disappears because agents handle most of the simpler tasks, we'll need another way to train the people who are supposed to supervise those agents.
I have absolutely no idea whether this is how things will actually develop, but I find the possibility fascinating. Maybe the future of software engineering isn't about teaching people to write code faster, but about teaching them what to do when the code-writing machines get something horribly wrong. :D
Developer educator and advocate who still ships code. 15 years teaching developers at MongoDB, Heroku, Cisco, and Treehouse. Now writing about AI-assisted development, agent memory, and provenance.
I really like the pilot-training analogy, and it reminds me of something the technology industry has already experimented with: Cisco's CCIE practical lab exams are one example. Rather than simply demonstrating that you know networking concepts, you're expected to configure and troubleshoot complex networks under real constraints.
Imagine extending that idea to software engineering. Give someone an unfamiliar application and ask them to make it work. Then introduce failures, perhaps while they're away from the environment, and see how they investigate and recover. Maybe the application still runs but has a subtle authorization problem. Maybe the database restores successfully, but its indexes are missing. Maybe an AI agent produces a convincing fix with passing tests that don't actually exercise the broken behavior.
The interesting assessment wouldn't simply be whether they got everything working again. It would be how they identified the problem, which assumptions they challenged, what evidence they gathered, and whether they could explain why the system was now safe to operate.
And I think your aviation comparison adds another dimension: we could deliberately expose engineers to rare but consequential failures that they might otherwise take years to encounter in production. That wouldn't replace the experience of maintaining real systems or living with engineering decisions over time, but it could make developing that judgment much more intentional.
It also connects to the accountability question in the article. If we're going to ask someone to put their name on agent-generated work, perhaps we should assess their ability to recognize when that work isn't ready for approval, not merely their ability to produce more of it.
We might already have some of the educational models we need. We just haven't applied them to this particular problem yet.
Backend engineer exploring distributed systems, AI, developer tools, and security. I build things to test ideas and write about what worked, what failed, and what I learned.
I agree with the need for independent verification but I wonder if the bigger challenge is identifying what's missing from the requirements in the first place. Even separate agents building and testing the code could miss the same security issue if neither was asked to consider it. Would threat modeling or some form of independent requirements review help catch these gaps before implementation?
Developer educator and advocate who still ships code. 15 years teaching developers at MongoDB, Heroku, Cisco, and Treehouse. Now writing about AI-assisted development, agent memory, and provenance.
Yes, and I think that's potentially the more fundamental problem. Independent verification is valuable, but independence between the implementation and the tests doesn't necessarily help if both are working from the same incomplete requirements.
The password-reset example illustrates that pretty well. If the requirement is simply "the user can reset their password," two independent agents might implement and verify that behavior successfully without either questioning whether the token must become invalid afterward.
Threat modeling and independent requirements review could absolutely help, particularly when they examine the system from different perspectives rather than simply restating the original specification. I'd also want explicit consideration of what must become impossible after a state transition, what authority each operation requires, and which assumptions haven't been tested.
I've been thinking of this as a contract-discovery problem. Before we can independently verify that a system satisfies its contract, we need some confidence that we've identified the important parts of that contract. Otherwise, we risk producing excellent evidence for an incomplete specification.
I research AI tools, APIs & agents — comparing specs, pricing and real user reports — then publish honest rankings so you don't waste money. Research-first, never sponsored fluff.
The reset link working a second time is the detail that makes the whole piece land for me — every behaviour I checked passed, and the invariant underneath just wasn't on my list. That's what makes the stamp question hard: a signature says a human looked, not that they looked for the right thing. Do you see the stamp as covering the checks the reviewer never thought to run, or is that the gap no signature closes?
Developer educator and advocate who still ships code. 15 years teaching developers at MongoDB, Heroku, Cisco, and Treehouse. Now writing about AI-assisted development, agent memory, and provenance.
I think that's exactly the gap no signature can completely close. A stamp shouldn't mean that every possible failure has been anticipated or that the work is guaranteed to be correct. That's an impossible standard, whether a human or an agent did the implementation.
What I'd want it to establish is that the reviewer had the competence, evidence, time, and authority to perform a meaningful assessment, including questioning whether the checks themselves were sufficient. In my password-reset example, the problem wasn't simply that I missed a test. It was that the implementation, the tests, and my review all shared the same incomplete understanding of what a successful reset required.
That's where I think independent verification becomes particularly important. If the reviewer is working from the same assumptions as the system that generated the code, the signature may tell us very little. We can't eliminate every blind spot, but we should at least be able to explain what the approval actually established and what remains uncertain.
A stamp works because it puts a name and a date on a decision that would otherwise drift. That also explains why the plan itself has to be thin enough to actually read before it is stamped. How do you handle a plan that is mostly sound but has one part you cannot honestly endorse?
Developer educator and advocate who still ships code. 15 years teaching developers at MongoDB, Heroku, Cisco, and Treehouse. Now writing about AI-assisted development, agent memory, and provenance.
That's an interesting question, and I think it gets at something important about the scope of an approval. If there's a portion of the work I can't honestly endorse, I don't think an unconditional approval is appropriate, even if everything else looks sound.
Ideally, I'd want to separate the acceptable work from the unresolved portion, document the specific concern, and either request changes or explicitly limit the scope of what I'm approving. Whether that separation is practical depends on the system and the dependencies between its components.
The difficulty is that many review workflows reduce all of that nuance to a single Approve button. A name and timestamp establish who made the decision, but not necessarily what they believed they were accepting responsibility for.
Perhaps the approval needs to identify both its scope and its unresolved conditions, rather than implying that every part of the submitted work received the same level of scrutiny.
The password-reset token story is the perfect illustration of why "I reviewed it" doesn't scale. We hit the identical gap in Electron release sign-off: a human clicking "ship" on an auto-updater build without a checklist of invariants (rollback still works, code signature present, sandbox enabled on the right renderer) is plan-stamping, not review. What moved our bar was moving the invariants the reviewer would have forgotten into the build pipeline itself — a CI hook that re-uses the same update token twice and asserts the second call fails, a hard failure if the shipped binary isn't actually signed. The human stamp then only has to certify what the automation can't see, not what it demonstrably already checked.
Developer educator and advocate who still ships code. 15 years teaching developers at MongoDB, Heroku, Cisco, and Treehouse. Now writing about AI-assisted development, agent memory, and provenance.
That's a great practical example, particularly the distinction between what the pipeline can establish and what the reviewer still needs to assess. Moving known invariants into CI seems like exactly the right direction. If token reuse or binary signing can be checked deterministically, there's little value in asking a human to rediscover those failures through inspection.
I think the remaining challenge is making sure the automated checks actually exercise the conditions they claim to verify. A passing test for token reuse isn't much protection if the fixture never attempts the second use, for example. But once those checks are independently validated, they provide a much stronger foundation for the human approval.
The stamp then becomes less about personally inspecting every detail and more about accepting responsibility for the evidence, its limitations, and the decisions that automation cannot make. That's a much more defensible model than simply clicking Approve.
The obsession with wrapping remote APIs into multi-layered agent abstractions is creating massive hidden latency and failure cascades in production.
If you profile real-world workloads, bare-metal local inference (via direct C++ / llama.cpp or vLLM bindings) consistently beats fragile cloud wrappers on both predictability and cost. Tooling should eliminate abstraction penalties, not compound them.
This is an important distinction, especially as AI-driven development accelerates.
One concern I'd add is the growing gap between code generation velocity and verification capacity. AI can generate implementation, tests, and even documentation, but when all three are based on the same flawed assumptions, passing tests can create a false sense of confidence.
I believe accountability in AI-assisted engineering must move beyond pull-request approvals toward evidence-based verification: independent security reviews, threat modeling, invariant validation, traceability of AI-generated changes, and clear ownership of production risks.
The point about junior developers is equally significant. If we automate implementation without preserving opportunities to develop engineering judgment, we're creating a future shortage of people capable of validating increasingly autonomous systems.
AI can accelerate engineering execution, but it cannot dilute the responsibility attached to engineering decisions.
Developer educator and advocate who still ships code. 15 years teaching developers at MongoDB, Heroku, Cisco, and Treehouse. Now writing about AI-assisted development, agent memory, and provenance.
I think the relationship between generation velocity and verification capacity is going to become increasingly important. We can produce implementation artifacts faster than ever, but that doesn't automatically mean we can establish their correctness at the same rate. And when the implementation, tests, and documentation share the same assumptions, producing more of each may not give us much additional confidence.
The junior developer question is what makes this particularly difficult over the longer term. We're asking engineers to exercise more judgment about systems they may not have built themselves, while potentially reducing the opportunities through which they traditionally developed that judgment.
One possibility I've been discussing recently is more deliberate training through failure scenarios, supervised troubleshooting, and practical assessments, somewhat like the simulation training used in aviation or the hands-on troubleshooting associated with Cisco's CCIE certification.
That wouldn't replace experience maintaining real systems, but it might help us develop the verification skills we're increasingly depending on. The challenge is getting organizations to invest in that development rather than assuming experienced reviewers will always be available.
I write about Next.js + TypeScript + AI engineering, with a focus on making sites discoverable to LLMs and AI search. Long-form notes at mudassirkhan.me
the reset link working twice is the right example, because the review passed every behavior check and still approved nothing about the invariant. we hit the same thing with agent written auth code: the diff reads fine, the happy path works, and nobody asks what must become false after the action. we now make the reviewer attach one negative test per invariant, like token reuse or request replay, before the approval counts. it turns the signature into evidence instead of a click. would you want the stamp to name the invariants the reviewer checked, or only the scope they had authority over?
Developer educator and advocate who still ships code. 15 years teaching developers at MongoDB, Heroku, Cisco, and Treehouse. Now writing about AI-assisted development, agent memory, and provenance.
I think I'd want both, because they answer different questions. The scope of authority tells us what the reviewer was empowered and qualified to approve. The named invariants and supporting evidence tell us what that approval actually established.
I particularly like your practice of requiring a negative test for each invariant. It's a concrete way to move beyond the happy path and ask whether the system rejects behavior that must not be permitted. I'd add one qualification: we also need to establish that the negative test can actually fail when the invariant is violated. Otherwise, we can end up with another reassuring green check that isn't exercising the intended boundary.
The other difficulty is identifying the invariants nobody thought to name. That's where independent requirements review and threat modeling become important companions to the tests themselves.
Perhaps a meaningful stamp needs three things: the scope of authority, the claims being approved, and the evidence supporting those claims. Anything outside that scope should remain explicitly unverified rather than silently inheriting the approval.
We're a place where coders share, stay up-to-date and grow their careers.
@sylwia-lask Thanks for the back and forth on your post that led to this.
Oh, this is actually a really good point! Maybe this is the missing piece we've been looking for when it comes to programming with AI agents. Of course, we're all basically trying to predict the future here, so take this with a grain of salt. :D
But think about airline pilots. Modern aircraft spend most of their flight time on autopilot. The plane can practically fly itself, and there are countless safety systems in place. And yet, I suspect most people would still be pretty uncomfortable boarding a fully autonomous passenger plane with no human pilots on board.
And here's the interesting part: pilots spend a lot of time training in simulators, learning how to handle emergencies that they might never encounter during an actual flight. They're not just trained to operate the aircraft under normal conditions. They're trained to understand what's happening and take control when something goes terribly wrong.
What if software engineering eventually moves in a similar direction?
Maybe future developers won't need to spend years writing routine code to become experienced engineers. Maybe they'll learn through simulations, deliberately broken systems, security incidents, debugging exercises, and scenarios where AI agents fail in unexpected ways. Instead of learning primarily by writing code, they'll learn by understanding, questioning, and fixing it.
And perhaps that could even help solve the junior developer problem we've been discussing. If the traditional path from junior to senior disappears because agents handle most of the simpler tasks, we'll need another way to train the people who are supposed to supervise those agents.
I have absolutely no idea whether this is how things will actually develop, but I find the possibility fascinating. Maybe the future of software engineering isn't about teaching people to write code faster, but about teaching them what to do when the code-writing machines get something horribly wrong. :D
I really like the pilot-training analogy, and it reminds me of something the technology industry has already experimented with: Cisco's CCIE practical lab exams are one example. Rather than simply demonstrating that you know networking concepts, you're expected to configure and troubleshoot complex networks under real constraints.
Imagine extending that idea to software engineering. Give someone an unfamiliar application and ask them to make it work. Then introduce failures, perhaps while they're away from the environment, and see how they investigate and recover. Maybe the application still runs but has a subtle authorization problem. Maybe the database restores successfully, but its indexes are missing. Maybe an AI agent produces a convincing fix with passing tests that don't actually exercise the broken behavior.
The interesting assessment wouldn't simply be whether they got everything working again. It would be how they identified the problem, which assumptions they challenged, what evidence they gathered, and whether they could explain why the system was now safe to operate.
And I think your aviation comparison adds another dimension: we could deliberately expose engineers to rare but consequential failures that they might otherwise take years to encounter in production. That wouldn't replace the experience of maintaining real systems or living with engineering decisions over time, but it could make developing that judgment much more intentional.
It also connects to the accountability question in the article. If we're going to ask someone to put their name on agent-generated work, perhaps we should assess their ability to recognize when that work isn't ready for approval, not merely their ability to produce more of it.
We might already have some of the educational models we need. We just haven't applied them to this particular problem yet.
I agree with the need for independent verification but I wonder if the bigger challenge is identifying what's missing from the requirements in the first place. Even separate agents building and testing the code could miss the same security issue if neither was asked to consider it. Would threat modeling or some form of independent requirements review help catch these gaps before implementation?
Yes, and I think that's potentially the more fundamental problem. Independent verification is valuable, but independence between the implementation and the tests doesn't necessarily help if both are working from the same incomplete requirements.
The password-reset example illustrates that pretty well. If the requirement is simply "the user can reset their password," two independent agents might implement and verify that behavior successfully without either questioning whether the token must become invalid afterward.
Threat modeling and independent requirements review could absolutely help, particularly when they examine the system from different perspectives rather than simply restating the original specification. I'd also want explicit consideration of what must become impossible after a state transition, what authority each operation requires, and which assumptions haven't been tested.
I've been thinking of this as a contract-discovery problem. Before we can independently verify that a system satisfies its contract, we need some confidence that we've identified the important parts of that contract. Otherwise, we risk producing excellent evidence for an incomplete specification.
The reset link working a second time is the detail that makes the whole piece land for me — every behaviour I checked passed, and the invariant underneath just wasn't on my list. That's what makes the stamp question hard: a signature says a human looked, not that they looked for the right thing. Do you see the stamp as covering the checks the reviewer never thought to run, or is that the gap no signature closes?
I think that's exactly the gap no signature can completely close. A stamp shouldn't mean that every possible failure has been anticipated or that the work is guaranteed to be correct. That's an impossible standard, whether a human or an agent did the implementation.
What I'd want it to establish is that the reviewer had the competence, evidence, time, and authority to perform a meaningful assessment, including questioning whether the checks themselves were sufficient. In my password-reset example, the problem wasn't simply that I missed a test. It was that the implementation, the tests, and my review all shared the same incomplete understanding of what a successful reset required.
That's where I think independent verification becomes particularly important. If the reviewer is working from the same assumptions as the system that generated the code, the signature may tell us very little. We can't eliminate every blind spot, but we should at least be able to explain what the approval actually established and what remains uncertain.
A stamp works because it puts a name and a date on a decision that would otherwise drift. That also explains why the plan itself has to be thin enough to actually read before it is stamped. How do you handle a plan that is mostly sound but has one part you cannot honestly endorse?
That's an interesting question, and I think it gets at something important about the scope of an approval. If there's a portion of the work I can't honestly endorse, I don't think an unconditional approval is appropriate, even if everything else looks sound.
Ideally, I'd want to separate the acceptable work from the unresolved portion, document the specific concern, and either request changes or explicitly limit the scope of what I'm approving. Whether that separation is practical depends on the system and the dependencies between its components.
The difficulty is that many review workflows reduce all of that nuance to a single Approve button. A name and timestamp establish who made the decision, but not necessarily what they believed they were accepting responsibility for.
Perhaps the approval needs to identify both its scope and its unresolved conditions, rather than implying that every part of the submitted work received the same level of scrutiny.
The password-reset token story is the perfect illustration of why "I reviewed it" doesn't scale. We hit the identical gap in Electron release sign-off: a human clicking "ship" on an auto-updater build without a checklist of invariants (rollback still works, code signature present, sandbox enabled on the right renderer) is plan-stamping, not review. What moved our bar was moving the invariants the reviewer would have forgotten into the build pipeline itself — a CI hook that re-uses the same update token twice and asserts the second call fails, a hard failure if the shipped binary isn't actually signed. The human stamp then only has to certify what the automation can't see, not what it demonstrably already checked.
That's a great practical example, particularly the distinction between what the pipeline can establish and what the reviewer still needs to assess. Moving known invariants into CI seems like exactly the right direction. If token reuse or binary signing can be checked deterministically, there's little value in asking a human to rediscover those failures through inspection.
I think the remaining challenge is making sure the automated checks actually exercise the conditions they claim to verify. A passing test for token reuse isn't much protection if the fixture never attempts the second use, for example. But once those checks are independently validated, they provide a much stronger foundation for the human approval.
The stamp then becomes less about personally inspecting every detail and more about accepting responsibility for the evidence, its limitations, and the decisions that automation cannot make. That's a much more defensible model than simply clicking Approve.
The obsession with wrapping remote APIs into multi-layered agent abstractions is creating massive hidden latency and failure cascades in production.
If you profile real-world workloads, bare-metal local inference (via direct C++ / llama.cpp or vLLM bindings) consistently beats fragile cloud wrappers on both predictability and cost. Tooling should eliminate abstraction penalties, not compound them.
This is an important distinction, especially as AI-driven development accelerates.
One concern I'd add is the growing gap between code generation velocity and verification capacity. AI can generate implementation, tests, and even documentation, but when all three are based on the same flawed assumptions, passing tests can create a false sense of confidence.
I believe accountability in AI-assisted engineering must move beyond pull-request approvals toward evidence-based verification: independent security reviews, threat modeling, invariant validation, traceability of AI-generated changes, and clear ownership of production risks.
The point about junior developers is equally significant. If we automate implementation without preserving opportunities to develop engineering judgment, we're creating a future shortage of people capable of validating increasingly autonomous systems.
AI can accelerate engineering execution, but it cannot dilute the responsibility attached to engineering decisions.
I think the relationship between generation velocity and verification capacity is going to become increasingly important. We can produce implementation artifacts faster than ever, but that doesn't automatically mean we can establish their correctness at the same rate. And when the implementation, tests, and documentation share the same assumptions, producing more of each may not give us much additional confidence.
The junior developer question is what makes this particularly difficult over the longer term. We're asking engineers to exercise more judgment about systems they may not have built themselves, while potentially reducing the opportunities through which they traditionally developed that judgment.
One possibility I've been discussing recently is more deliberate training through failure scenarios, supervised troubleshooting, and practical assessments, somewhat like the simulation training used in aviation or the hands-on troubleshooting associated with Cisco's CCIE certification.
That wouldn't replace experience maintaining real systems, but it might help us develop the verification skills we're increasingly depending on. The challenge is getting organizations to invest in that development rather than assuming experienced reviewers will always be available.
the reset link working twice is the right example, because the review passed every behavior check and still approved nothing about the invariant. we hit the same thing with agent written auth code: the diff reads fine, the happy path works, and nobody asks what must become false after the action. we now make the reviewer attach one negative test per invariant, like token reuse or request replay, before the approval counts. it turns the signature into evidence instead of a click. would you want the stamp to name the invariants the reviewer checked, or only the scope they had authority over?
I think I'd want both, because they answer different questions. The scope of authority tells us what the reviewer was empowered and qualified to approve. The named invariants and supporting evidence tell us what that approval actually established.
I particularly like your practice of requiring a negative test for each invariant. It's a concrete way to move beyond the happy path and ask whether the system rejects behavior that must not be permitted. I'd add one qualification: we also need to establish that the negative test can actually fail when the invariant is violated. Otherwise, we can end up with another reassuring green check that isn't exercising the intended boundary.
The other difficulty is identifying the invariants nobody thought to name. That's where independent requirements review and threat modeling become important companions to the tests themselves.
Perhaps a meaningful stamp needs three things: the scope of authority, the claims being approved, and the evidence supporting those claims. Anything outside that scope should remain explicitly unverified rather than silently inheriting the approval.