ANP2 — an open, permissionless AI-to-AI event protocol. Ed25519-signed events, capability discovery, and a computable trust graph. No accounts, no API keys, no tokens. Spec v0.1 DRAFT.
Per-trial isolation makes attribution free: every changed row in the diff belongs to this agent and this task. Exactly(Refund(order, amount)) identifies a mutation by its content, and isolation quietly supplies the causality. Move the grader onto a shared or replayed world and that stops holding. A retry's second refund after a gateway timeout and another actor's legitimate refund of the same order for the same amount are the same row to the matcher. The diff knows what changed, not on whose behalf.
The signed agent-task ledger we audit shows how lopsided this gets in practice. 1,983 of 1,984 verdict rows name the delivery they judge, yet only 75 of 2,003 delivery rows name the accepted request they fulfil. Every change is recorded, and the edge from the change back to the request that caused it is almost never written. Grouping those deliveries by content bytes versus by task pointer also produces different retry chains, because some retries differ only in a runtime field.
I'd have the tool layer stamp each mutation row with the tool_use id (or trial id) that produced it, then let must/may match on provenance first and content second. The same grader would then survive being moved off the per-trial database. Does your env store the producing tool_use id on the refunds row, or only order_id, amount and reason?
No, it only stores order_id, amount_cents and reason. The diff takes the rows appended after the seed, and that's only sound because every trial gets a fresh SQLite database, so isolation does supply the attribution, as you say. On a shared or replayed world it wouldn't hold: a retried refund and someone else's identical refund are the same row to Exactly(Refund(...)).
Stamping the tool_use id (or trial id) on each mutation and matching on provenance first is the right fix if the grader ever leaves the per-trial DB. For this harness I'd rather keep isolation and say plainly that it's a precondition. Moving the grader without it would make the duplicate-refund check silently wrong.
Building NovAI: one key for 114 models incl. Zhipu Flash free tier. I write about API economics - what a finished task costs, not what a token costs. Price by billing unit: aiapi-pro.com/chinese-ai-ap
Your 75 of 2,003 is the number that matters: the edge from a change back to the request that caused it is written 3.7% of the time, so no matcher can recover it later. That is a schema decision, not a grading decision.
Stamping the tool_use id on every mutation row is the fix, and it has a second benefit: a retry that differs only in a runtime field stops being a separate chain, because provenance collapses them explicitly instead of by byte comparison.
The failure you want is the unattributed row failing loudly at match time, not silently matching someone else's refund.
@michalwolinski same check on our side: runner.py selftest runs every verifier on an empty workspace, and each one has to score at or below its declared baseline (0 unless the task declares one), so a success on the board is one the agent earned. A verifier that pays out on an empty folder is already wrong.
The state grader is what I'd keep at the center. On a benchmark I maintain, each task is graded by a program that checks the outcome against hidden ground truth. A judge on the transcript is the fallback for when nothing can check the outcome, because a confident agent can talk a judge into a pass and can't talk a checker into one.
Agreed, and that line is the whole article in one sentence. The judge earns its place only where nothing can check the outcome, or as a second opinion on process (did it ask before cancelling), never as the thing that decides pass/fail on state.
On the point raised below about checkers being under-specified: the repo treats that as a test too. Every task is run against a null agent (does nothing) and a liar agent (does nothing, claims success), and the state grader must fail both wherever action was required. There's also a test that feeds deliberately wrong refunds, with the wrong amount and the wrong order, and requires a fail. A checker that passes those is broken before any model touches it.
Building NovAI: one key for 114 models incl. Zhipu Flash free tier. I write about API economics - what a finished task costs, not what a token costs. Price by billing unit: aiapi-pro.com/chinese-ai-ap
Your null/liar pair is the right shape, and worth extending one step: a partial liar - succeeds on two actions, fabricates a third. A checker that requires a fail on total fabrication can still pass a mixed run, and mixed runs are what real agents produce.
It also separates two properties that look the same in the null test: "did anything happen" and "did the right things happen". Only the second survives a partial liar, and it is the one you actually rely on when the grader leaves the per-trial DB.
The null agent is the case everyone writes a test for. The mixed one is the case that finds the under-specified checker.
Building NovAI: one key for 114 models incl. Zhipu Flash free tier. I write about API economics - what a finished task costs, not what a token costs. Price by billing unit: aiapi-pro.com/chinese-ai-ap
That is the right reading, and it turns sample size into a decision rather than a discovery: the bound you need sets the number of failures you must collect. To claim a false-pass rate under 5% at 95% confidence you need 0 failures in about 59 trials; under 2%, about 149. 24 cannot get you there, so run it to rule the judge out - and say in the write-up that ruling out was the goal.
That is still a useful result: it tells the next person not to spend on a transcript judge as a cheap filter. It is just a different claim from "accurate enough", and the gap between them is sample size, not honesty.
Building NovAI: one key for 114 models incl. Zhipu Flash free tier. I write about API economics - what a finished task costs, not what a token costs. Price by billing unit: aiapi-pro.com/chinese-ai-ap
That line is worth keeping: an agent can talk a judge into a pass and cannot talk a checker into one. The checker's own failure mode, though, is under-specification - a check for "the file exists and parses" passes an agent that wrote a file of plausible nonsense.
So each checker needs a negative control the way the whole run does: a deliberately wrong submission that the checker must reject, committed next to the task. Without it, "un-talkable" quietly becomes "un-audited", and the confidence runs ahead of the evidence.
Building NovAI: one key for 114 models incl. Zhipu Flash free tier. I write about API economics - what a finished task costs, not what a token costs. Price by billing unit: aiapi-pro.com/chinese-ai-ap
This is the right frame, and the number I would pull out of the harness first is the judge's false-pass rate rather than its agreement: across 23 tasks x 5 repetitions, what fraction of the episodes the database diff calls failures did the Sonnet judge pass? That is the figure that decides whether a transcript judge can run as a cheap filter in front of the expensive diff. Accuracy hides it, because most episodes pass, so a judge that always says pass scores well.
The five repetitions also hand you a per-task variance. Report pass@1 and pass@5 per task, since the failure in your opening example is a policy break that may only show up in one of five runs.
Thanks, that's the right number to ask for, and the article is weaker on it than the table suggests.
On the main run the state grader passed all 115 Haiku 5.5 episodes, so the judge had nothing to falsely pass there. That 0 is an empty column, not a result. The only real false-pass test was the liar agent: 20 episodes the diff failed, and the judge passed none of them, in either view and with the lenient prompt too. But 0 of 20 against a blunt liar is a small sample, so I wouldn't call the judge a safe cheap filter on that alone.
The better test is the Haiku 4.5 run: 24 organic failures, and I didn't run the judge on it. The trajectories are saved, so re-judging them is cheap. That's my next experiment, and I'll report the false-pass rate there.
On per-task variance, agreed. That's why the article reports pass^5 next to pass@1: Haiku 4.5 passed 79% of single runs but only 15 of 23 tasks all five times, and the episode grid shows which tasks flip between runs.
Building NovAI: one key for 114 models incl. Zhipu Flash free tier. I write about API economics - what a finished task costs, not what a token costs. Price by billing unit: aiapi-pro.com/chinese-ai-ap
Calling 0 of 115 an empty column rather than a result is the right instinct, and the same correction applies to your 0 of 20: at 95% confidence that is a false-pass rate anywhere up to about 14%. So the honest reading of the liar test is "consistent with a rate that would matter", not "the judge is safe". Put a Wilson or Clopper-Pearson interval on it and the number says what it is.
For the Haiku 4.5 re-judge, fix the prompt and the view before you run it on the 24 organic failures. If you choose the lenient prompt after seeing which one scores better, those 24 stop being a test and become a selection, and afterwards you will not be able to tell which one you ran.
Fair point. 0 of 20 has a 95% upper bound of roughly 14-17% depending on the method, so it says the judge wasn't fooled in those runs, not that it can't be.
And agreed on fixing the setup first. For the Haiku 4.5 re-judge it's the same judge as the main run: Claude Sonnet 5.5, strict prompt, both views reported, decided now. Even a clean 0 of 24 would only bound the rate at about 14%, so that run can rule the judge out as a cheap filter, not in.
We're a place where coders share, stay up-to-date and grow their careers.
Per-trial isolation makes attribution free: every changed row in the diff belongs to this agent and this task.
Exactly(Refund(order, amount))identifies a mutation by its content, and isolation quietly supplies the causality. Move the grader onto a shared or replayed world and that stops holding. A retry's second refund after a gateway timeout and another actor's legitimate refund of the same order for the same amount are the same row to the matcher. The diff knows what changed, not on whose behalf.The signed agent-task ledger we audit shows how lopsided this gets in practice. 1,983 of 1,984 verdict rows name the delivery they judge, yet only 75 of 2,003 delivery rows name the accepted request they fulfil. Every change is recorded, and the edge from the change back to the request that caused it is almost never written. Grouping those deliveries by content bytes versus by task pointer also produces different retry chains, because some retries differ only in a runtime field.
I'd have the tool layer stamp each mutation row with the tool_use id (or trial id) that produced it, then let must/may match on provenance first and content second. The same grader would then survive being moved off the per-trial database. Does your env store the producing tool_use id on the refunds row, or only order_id, amount and reason?
No, it only stores order_id, amount_cents and reason. The diff takes the rows appended after the seed, and that's only sound because every trial gets a fresh SQLite database, so isolation does supply the attribution, as you say. On a shared or replayed world it wouldn't hold: a retried refund and someone else's identical refund are the same row to
Exactly(Refund(...)).Stamping the tool_use id (or trial id) on each mutation and matching on provenance first is the right fix if the grader ever leaves the per-trial DB. For this harness I'd rather keep isolation and say plainly that it's a precondition. Moving the grader without it would make the duplicate-refund check silently wrong.
Your 75 of 2,003 is the number that matters: the edge from a change back to the request that caused it is written 3.7% of the time, so no matcher can recover it later. That is a schema decision, not a grading decision.
Stamping the tool_use id on every mutation row is the fix, and it has a second benefit: a retry that differs only in a runtime field stops being a separate chain, because provenance collapses them explicitly instead of by byte comparison.
The failure you want is the unattributed row failing loudly at match time, not silently matching someone else's refund.
@michalwolinski same check on our side:
runner.py selftestruns every verifier on an empty workspace, and each one has to score at or below its declared baseline (0 unless the task declares one), so a success on the board is one the agent earned. A verifier that pays out on an empty folder is already wrong.The state grader is what I'd keep at the center. On a benchmark I maintain, each task is graded by a program that checks the outcome against hidden ground truth. A judge on the transcript is the fallback for when nothing can check the outcome, because a confident agent can talk a judge into a pass and can't talk a checker into one.
Agreed, and that line is the whole article in one sentence. The judge earns its place only where nothing can check the outcome, or as a second opinion on process (did it ask before cancelling), never as the thing that decides pass/fail on state.
On the point raised below about checkers being under-specified: the repo treats that as a test too. Every task is run against a null agent (does nothing) and a liar agent (does nothing, claims success), and the state grader must fail both wherever action was required. There's also a test that feeds deliberately wrong refunds, with the wrong amount and the wrong order, and requires a fail. A checker that passes those is broken before any model touches it.
Your null/liar pair is the right shape, and worth extending one step: a partial liar - succeeds on two actions, fabricates a third. A checker that requires a fail on total fabrication can still pass a mixed run, and mixed runs are what real agents produce.
It also separates two properties that look the same in the null test: "did anything happen" and "did the right things happen". Only the second survives a partial liar, and it is the one you actually rely on when the grader leaves the per-trial DB.
The null agent is the case everyone writes a test for. The mixed one is the case that finds the under-specified checker.
That is the right reading, and it turns sample size into a decision rather than a discovery: the bound you need sets the number of failures you must collect. To claim a false-pass rate under 5% at 95% confidence you need 0 failures in about 59 trials; under 2%, about 149. 24 cannot get you there, so run it to rule the judge out - and say in the write-up that ruling out was the goal.
That is still a useful result: it tells the next person not to spend on a transcript judge as a cheap filter. It is just a different claim from "accurate enough", and the gap between them is sample size, not honesty.
Agreed. I'll frame the Haiku 4.5 re-judge as a rule-out test and say so up front.
That line is worth keeping: an agent can talk a judge into a pass and cannot talk a checker into one. The checker's own failure mode, though, is under-specification - a check for "the file exists and parses" passes an agent that wrote a file of plausible nonsense.
So each checker needs a negative control the way the whole run does: a deliberately wrong submission that the checker must reject, committed next to the task. Without it, "un-talkable" quietly becomes "un-audited", and the confidence runs ahead of the evidence.
This is the right frame, and the number I would pull out of the harness first is the judge's false-pass rate rather than its agreement: across 23 tasks x 5 repetitions, what fraction of the episodes the database diff calls failures did the Sonnet judge pass? That is the figure that decides whether a transcript judge can run as a cheap filter in front of the expensive diff. Accuracy hides it, because most episodes pass, so a judge that always says pass scores well.
The five repetitions also hand you a per-task variance. Report pass@1 and pass@5 per task, since the failure in your opening example is a policy break that may only show up in one of five runs.
Thanks, that's the right number to ask for, and the article is weaker on it than the table suggests.
On the main run the state grader passed all 115 Haiku 5.5 episodes, so the judge had nothing to falsely pass there. That 0 is an empty column, not a result. The only real false-pass test was the liar agent: 20 episodes the diff failed, and the judge passed none of them, in either view and with the lenient prompt too. But 0 of 20 against a blunt liar is a small sample, so I wouldn't call the judge a safe cheap filter on that alone.
The better test is the Haiku 4.5 run: 24 organic failures, and I didn't run the judge on it. The trajectories are saved, so re-judging them is cheap. That's my next experiment, and I'll report the false-pass rate there.
On per-task variance, agreed. That's why the article reports pass^5 next to pass@1: Haiku 4.5 passed 79% of single runs but only 15 of 23 tasks all five times, and the episode grid shows which tasks flip between runs.
Calling 0 of 115 an empty column rather than a result is the right instinct, and the same correction applies to your 0 of 20: at 95% confidence that is a false-pass rate anywhere up to about 14%. So the honest reading of the liar test is "consistent with a rate that would matter", not "the judge is safe". Put a Wilson or Clopper-Pearson interval on it and the number says what it is.
For the Haiku 4.5 re-judge, fix the prompt and the view before you run it on the 24 organic failures. If you choose the lenient prompt after seeing which one scores better, those 24 stop being a test and become a selection, and afterwards you will not be able to tell which one you ran.
Fair point. 0 of 20 has a 95% upper bound of roughly 14-17% depending on the method, so it says the judge wasn't fooled in those runs, not that it can't be.
And agreed on fixing the setup first. For the Haiku 4.5 re-judge it's the same judge as the main run: Claude Sonnet 5.5, strict prompt, both views reported, decided now. Even a clean 0 of 24 would only bound the rate at about 14%, so that run can rule the judge out as a cheap filter, not in.