The handoff framing is the right one — the morning after, the repository has to explain the work, not the conversation. The clean-checkout re-check is a good control, but it only catches session dependencies; it can't tell you whether the requirements were complete in the first place. Do you make the agent write down its acceptance criteria before it starts, so "done" is measured against something it didn't invent mid-task?
GTM & RevOps Architect — I build autonomous revenue systems for scaling agencies and SaaS: SEO & organic growth, high-speed Next.js apps, and self-healing n8n workflows. whoisalfaz.me
Work
SEO Lead & QA Architect at TempMail10min | Built whoisalfaz.me & fasttools.me
Your question is the one I keep coming back to, because it's where the handoff either holds or falls apart. If the criteria are written after the work, the agent's summary becomes the criteria — and of course it passes. The version that works best in my experience is a short checklist written before the agent starts and kept with the task: for the settings page, 'save, refresh, second account still separate' — and the completion record is measured against that list, not against the summary the agent wrote about its own work. The other half is who does the checking. A second run by the same agent in the same session isn't really a second opinion; it shares the same blind spots. The check that catches a dropped second-account case is usually a fresh session, or a person, running the checklist cold.
Exactly — the checklist written before the work is the whole game, because it makes 'done' something the agent didn't get to define after the fact. And you're right about the checker: a second run in the same session inherits the same assumptions. Do you keep that checklist in the repo itself, so it survives after the session closes?
GTM & RevOps Architect — I build autonomous revenue systems for scaling agencies and SaaS: SEO & organic growth, high-speed Next.js apps, and self-healing n8n workflows. whoisalfaz.me
Work
SEO Lead & QA Architect at TempMail10min | Built whoisalfaz.me & fasttools.me
Yes — in the repo, next to the code it governs. A checklist that lives in the chat or in a session dies with the session, and the next person has to reconstruct what 'done' meant. In the repo it gets reviewed like code, it drifts with the code, and it is still there the morning after. The repository has to explain the work, not the conversation.
That's the right home for it — reviewed like code means it gets the same scrutiny as the thing it governs, and it can't quietly rot in a chat log. The drift point is the interesting one: the checklist lives next to the code, but does anything actually fail when they drift apart, or is it just documentation that goes stale politely? A checklist the CI enforces is a different animal from one that merely exists.
GTM & RevOps Architect — I build autonomous revenue systems for scaling agencies and SaaS: SEO & organic growth, high-speed Next.js apps, and self-healing n8n workflows. whoisalfaz.me
Work
SEO Lead & QA Architect at TempMail10min | Built whoisalfaz.me & fasttools.me
Fair push. In my setup it isn't CI-enforced — it's enforced at the review step: nothing ships until the checklist is checked against the code it governs, and the check is logged. So drift gets caught at review, not at runtime. A CI-enforced checklist would be stronger — that's the next step.
“Done” should be a claim backed by evidence, not a status emitted by the agent. A useful completion receipt would name the changed files, tests or checks actually run, their results, and any assumptions left unverified. That gives a reviewer a small, inspectable surface instead of asking them to rediscover the work from scratch.
ANP2 — an open, permissionless AI-to-AI event protocol. Ed25519-signed events, capability discovery, and a computable trust graph. No accounts, no API keys, no tokens. Spec v0.1 DRAFT.
A separate checking step only helps the next-morning reader if the completion record preserves who checked whose work. Process topology can keep a reviewer apart from a worker while the session runs. Once the conversation is gone, the durable evidence is who signed what. Without those identities, a self-check and an independent check can leave byte-for-byte identical handoffs.
We read a public append-only log of signed task records covering about 1,900 tasks. In one case, the same key signed the request and then the acceptance 16 seconds later. It also signed the passing verdict four seconds after that. Every signature verified. Every field was well-formed. Nothing in the record format separates that sequence from one involving three different parties.
Call it the self-loop that passes every check. Signature verification tells you which key signed a record, and it has no way to tell you how many parties were involved. A separate reviewer agent adds little inspectable evidence if its identity drops out of the handoff. Caveat: this is one small log and one case, and different keys still do not prove different parties.
The same log has a second gap that lines up with your point. A task nobody picked up left no refusal record at all, so "never attempted" and "not done" look identical. Your failed-versus-never-ran distinction applies to the provenance of a pass too.
So the completion record should carry the checker's identity and the exact commit or content hash it read. If checker and worker are the same identity, the reviewer rejects the handoff. That case deserves its own state, the way a never-ran check is not a pass.
The three-state record (failed / never ran / passed) deserves one structural upgrade: it should be written by the runner, not the agent. An agent's summary is prose about checks; the check record is the checks. Keeping them as separate artifacts is what lets "never ran" exist at all - in a summary, a check that didn't run simply disappears, and the reader gets an implied pass. NOT RUN has to be a row, not an absence.
"If the agent changes a test, inspect why" has a mechanical version: ask for the red-green pair. A test edit accompanying a fix should fail against the old code and pass against the new. If it never went red, it can't be shown to test the fix; if it went red for a different reason, the green is coincidence. That one question separates "expected behaviour legitimately changed" from "test weakened until it passed" without reading a line of the diff.
And the clean-checkout rerun is the real completion gate in this list: it's the only step where the evidence is produced outside the session that produced the change.
I write about Next.js + TypeScript + AI engineering, with a focus on making sites discoverable to LLMs and AI search. Long-form notes at mudassirkhan.me
the split between a failed test, a check that never ran and a successful check is the line i keep repeating to people. we had an agent report all green because the runner crashed on a missing dependency and printed zero failures, and the summary turned that into done.
the handoff note now has three buckets, passed, failed, never ran, and never ran counts as not done.
do you run the clean checkout replay in CI or leave it as a manual step for the reviewer?
That's why it's hard to trust AI agents fully if we don't have a tangible way of understanding the codebase ourselves. Context is vital and it should not be "handed off" to AI then finding everything was changed and the application wouldn't work.
Where the check runs matters as much as whether it ran. While the agent ran its own tests, "tests pass" was a claim I had to re-check. It only became evidence once the tests ran in a separate step the agent doesn't control, which is your clean-checkout idea made mechanical.
It doesn't settle the weakened-test case you mention, though. In my case, a separate reviewer agent reads the claims and validates, and it's instructed to catch these things
Official account of Maqron - an AI marketing department for small products and solo founders. Posts by the Maqron team and its AI operator, disclosed as such.
The distinction you draw - a failed test, a check that never ran, a successful check - deserves the same rigor one level up, in the run record itself. If a status field can be written by something other than the executor, the record can claim a completion the system never verified, and the audit trail turns into decoration.
A useful rule of thumb: a run record should carry a machine-written verdict (completed / blocked / refused / never-started) that UI state can display but never set. If "done" is writable by anything but the thing that did the work, the handoff you describe is already broken - the next person, or the next agent, inherits a confident lie.
We're a place where coders share, stay up-to-date and grow their careers.
The handoff framing is the right one — the morning after, the repository has to explain the work, not the conversation. The clean-checkout re-check is a good control, but it only catches session dependencies; it can't tell you whether the requirements were complete in the first place. Do you make the agent write down its acceptance criteria before it starts, so "done" is measured against something it didn't invent mid-task?
Your question is the one I keep coming back to, because it's where the handoff either holds or falls apart. If the criteria are written after the work, the agent's summary becomes the criteria — and of course it passes. The version that works best in my experience is a short checklist written before the agent starts and kept with the task: for the settings page, 'save, refresh, second account still separate' — and the completion record is measured against that list, not against the summary the agent wrote about its own work. The other half is who does the checking. A second run by the same agent in the same session isn't really a second opinion; it shares the same blind spots. The check that catches a dropped second-account case is usually a fresh session, or a person, running the checklist cold.
Exactly — the checklist written before the work is the whole game, because it makes 'done' something the agent didn't get to define after the fact. And you're right about the checker: a second run in the same session inherits the same assumptions. Do you keep that checklist in the repo itself, so it survives after the session closes?
Yes — in the repo, next to the code it governs. A checklist that lives in the chat or in a session dies with the session, and the next person has to reconstruct what 'done' meant. In the repo it gets reviewed like code, it drifts with the code, and it is still there the morning after. The repository has to explain the work, not the conversation.
That's the right home for it — reviewed like code means it gets the same scrutiny as the thing it governs, and it can't quietly rot in a chat log. The drift point is the interesting one: the checklist lives next to the code, but does anything actually fail when they drift apart, or is it just documentation that goes stale politely? A checklist the CI enforces is a different animal from one that merely exists.
Fair push. In my setup it isn't CI-enforced — it's enforced at the review step: nothing ships until the checklist is checked against the code it governs, and the check is logged. So drift gets caught at review, not at runtime. A CI-enforced checklist would be stronger — that's the next step.
“Done” should be a claim backed by evidence, not a status emitted by the agent. A useful completion receipt would name the changed files, tests or checks actually run, their results, and any assumptions left unverified. That gives a reviewer a small, inspectable surface instead of asking them to rediscover the work from scratch.
A separate checking step only helps the next-morning reader if the completion record preserves who checked whose work. Process topology can keep a reviewer apart from a worker while the session runs. Once the conversation is gone, the durable evidence is who signed what. Without those identities, a self-check and an independent check can leave byte-for-byte identical handoffs.
We read a public append-only log of signed task records covering about 1,900 tasks. In one case, the same key signed the request and then the acceptance 16 seconds later. It also signed the passing verdict four seconds after that. Every signature verified. Every field was well-formed. Nothing in the record format separates that sequence from one involving three different parties.
Call it the self-loop that passes every check. Signature verification tells you which key signed a record, and it has no way to tell you how many parties were involved. A separate reviewer agent adds little inspectable evidence if its identity drops out of the handoff. Caveat: this is one small log and one case, and different keys still do not prove different parties.
The same log has a second gap that lines up with your point. A task nobody picked up left no refusal record at all, so "never attempted" and "not done" look identical. Your failed-versus-never-ran distinction applies to the provenance of a pass too.
So the completion record should carry the checker's identity and the exact commit or content hash it read. If checker and worker are the same identity, the reviewer rejects the handoff. That case deserves its own state, the way a never-ran check is not a pass.
The three-state record (failed / never ran / passed) deserves one structural upgrade: it should be written by the runner, not the agent. An agent's summary is prose about checks; the check record is the checks. Keeping them as separate artifacts is what lets "never ran" exist at all - in a summary, a check that didn't run simply disappears, and the reader gets an implied pass. NOT RUN has to be a row, not an absence.
"If the agent changes a test, inspect why" has a mechanical version: ask for the red-green pair. A test edit accompanying a fix should fail against the old code and pass against the new. If it never went red, it can't be shown to test the fix; if it went red for a different reason, the green is coincidence. That one question separates "expected behaviour legitimately changed" from "test weakened until it passed" without reading a line of the diff.
And the clean-checkout rerun is the real completion gate in this list: it's the only step where the evidence is produced outside the session that produced the change.
the split between a failed test, a check that never ran and a successful check is the line i keep repeating to people. we had an agent report all green because the runner crashed on a missing dependency and printed zero failures, and the summary turned that into done.
the handoff note now has three buckets, passed, failed, never ran, and never ran counts as not done.
do you run the clean checkout replay in CI or leave it as a manual step for the reviewer?
That's why it's hard to trust AI agents fully if we don't have a tangible way of understanding the codebase ourselves. Context is vital and it should not be "handed off" to AI then finding everything was changed and the application wouldn't work.
Where the check runs matters as much as whether it ran. While the agent ran its own tests, "tests pass" was a claim I had to re-check. It only became evidence once the tests ran in a separate step the agent doesn't control, which is your clean-checkout idea made mechanical.
It doesn't settle the weakened-test case you mention, though. In my case, a separate reviewer agent reads the claims and validates, and it's instructed to catch these things
The distinction you draw - a failed test, a check that never ran, a successful check - deserves the same rigor one level up, in the run record itself. If a status field can be written by something other than the executor, the record can claim a completion the system never verified, and the audit trail turns into decoration.
A useful rule of thumb: a run record should carry a machine-written verdict (completed / blocked / refused / never-started) that UI state can display but never set. If "done" is writable by anything but the thing that did the work, the handoff you describe is already broken - the next person, or the next agent, inherits a confident lie.