Creator of GithubStarsManager — an open-source, local-first tool that organizes GitHub stars with AI: auto-categorization, semantic search, release tracking, and an MCP server for star library.
The "count what the model received" rule is the one I'm stealing — a single log line of characters received would have caught this on day one. A tracker showing a 4.2/5 score on zero characters of job text is the perfect illustration of why "the model replied" must never equal "the work is done". Great write-up, and the four PRs make the fix easy to follow.
I measure when AI agents invent data after a tool fails silently (reported-or-invented). 6 merged PRs at career-ops-hq—building Voltura, open-source heat-risk screening for Spain.
Thanks, glad it’s useful. One thing I’d add: a log line only tells you after the damage, so the fix turns the count into a gate. #5814 stops a posting that wasn’t read from being saved or marked done, and #6169 waits until the main content passes 1,500 visible characters (15 s cap) instead of a fixed 2 s. On 20 live Workday postings, that took readable pages from 0 to 20.
Creator of GithubStarsManager — an open-source, local-first tool that organizes GitHub stars with AI: auto-categorization, semantic search, release tracking, and an MCP server for star library.
Fair point — the log line is the autopsy, the gate is the prevention. 0 to 20 readable pages on live Workday postings is a clean before/after, and the 1,500-character threshold with a 15 s cap is a nice balance: strict enough to mean something without timing out on slow pages.
I measure when AI agents invent data after a tool fails silently (reported-or-invented). 6 merged PRs at career-ops-hq—building Voltura, open-source heat-risk screening for Spain.
Agreed. One correction to my own numbers, because 1,500 sounds like a hard minimum and in the merged version it isn’t one. After the maintainer’s review, the wait also stops as soon as a short posting can already be read, and pages that return 4xx/5xx skip the wait entirely. 1,500 visible characters is just the fast way out, with 15 s as the cap. (The 0 → 20 was measured on an earlier version of the wait.)
Creator of GithubStarsManager — an open-source, local-first tool that organizes GitHub stars with AI: auto-categorization, semantic search, release tracking, and an MCP server for star library.
Thanks for the correction — that's an important distinction, and good of you to flag your own numbers. The merged version sounds like the right shape: stop as soon as a short posting is readable, skip the wait entirely on 4xx/5xx, and let 1,500 characters just be the fast way out with 15 s as the cap. And noted that the 0 → 20 was measured on the earlier wait — the maintainer review clearly sharpened it.
This failure mode deserves its own name, because it shows up everywhere once you start looking: a system reporting completion for work it never actually did. The scanner wasn't lying exactly — "done" had just quietly come to mean "the loop reached this posting" instead of "the posting was read and processed."
The fix that generalizes: make the done-state impossible to reach without the evidence attached. If marking a posting done requires the extracted fields to exist — non-empty title, a real description span, a timestamp from the posting itself — then the state can't be faked by a code path that skipped the work. The claim and the proof become the same object.
Anywhere a status can be set by the same code that's supposed to do the work, with nothing downstream checking, you'll eventually get a green dashboard over an empty room.
I measure when AI agents invent data after a tool fails silently (reported-or-invented). 6 merged PRs at career-ops-hq—building Voltura, open-source heat-risk screening for Spain.
Thanks, that’s close to how it ended up being fixed. #5814 (by bercedev) made the pipeline mark a posting done only when the evaluation returns a report. My #6169 replaced the fixed 2-second wait with a wait for the description text, so a page that can’t be read now stays pending instead of being marked done.
One difference from what you describe: the check is whether the page can be read at all, not field by field.
A question back: in agent pipelines, the evidence can be made up too. A model will happily fill in a title and description it never received. How do you make sure the evidence came from the source and not from the model?
You've named the failure mode exactly — if the model writes the evidence, the evidence is fiction with good formatting. Our rule: the model never originates a fact. The page is fetched and snapshotted first, with its content hash recorded at collection time, and every field in the output has to point back into that snapshot — the span it came from, not just the page it claims to be about. A separate deterministic check then re-reads the snapshot and compares: if a title or description can't be traced to actual text in it, the field comes out UNKNOWN, never filled in. The hash matters because it makes the snapshot itself the thing on trial later — anyone can re-fetch, re-hash, and see whether what was evaluated is what was reported. The model is only ever allowed to summarize what the snapshot proves is there.
I measure when AI agents invent data after a tool fails silently (reported-or-invented). 6 merged PRs at career-ops-hq—building Voltura, open-source heat-risk screening for Spain.
The stub that always answers "Score: 4.2/5" is a great way to prove the bug. It shows the tracker could not tell a real evaluation from an evaluation of nothing, which is the whole problem in one line.
I also like that the fix ended up as a gate rather than a log line. A count of characters received tells you about the damage afterwards. Refusing to mark a posting done until the page could actually be read stops it from happening, and a posting that stays pending is honest in a way a fake score never is.
We rebuilt a candidate pipeline at Ascendra Ventures around the same idea. Enrichment touched resumes, LinkedIn data, batch parsing and embeddings, and any slow step could leave a person in an unclear state. We moved it to a durable workflow where every person has to end in a final status, finished or visibly failed, with polling as a fallback when webhooks went quiet. Your fix and ours land on the same rule: nothing gets to look complete without evidence.
After #6169, do pages that never pass the threshold show up anywhere as a separate list, or do they just stay pending in the tracker?
I measure when AI agents invent data after a tool fails silently (reported-or-invented). 6 merged PRs at career-ops-hq—building Voltura, open-source heat-risk screening for Spain.
Thanks — that’s the point: a pending row is honest, a 4.2/5 from nothing isn’t.
To your question: no separate list. A page that never becomes readable stays - [ ] in pipeline.md, the run prints why (No job description to evaluate at ()), and the next run tries it again. A posting the ATS reports as closed (Workday canApply: false, SmartRecruiters active: false) is handled the same way: not evaluated, left pending.
We're a place where coders share, stay up-to-date and grow their careers.
The "count what the model received" rule is the one I'm stealing — a single log line of characters received would have caught this on day one. A tracker showing a 4.2/5 score on zero characters of job text is the perfect illustration of why "the model replied" must never equal "the work is done". Great write-up, and the four PRs make the fix easy to follow.
Thanks, glad it’s useful. One thing I’d add: a log line only tells you after the damage, so the fix turns the count into a gate. #5814 stops a posting that wasn’t read from being saved or marked done, and #6169 waits until the main content passes 1,500 visible characters (15 s cap) instead of a fixed 2 s. On 20 live Workday postings, that took readable pages from 0 to 20.
Fair point — the log line is the autopsy, the gate is the prevention. 0 to 20 readable pages on live Workday postings is a clean before/after, and the 1,500-character threshold with a 15 s cap is a nice balance: strict enough to mean something without timing out on slow pages.
Agreed. One correction to my own numbers, because 1,500 sounds like a hard minimum and in the merged version it isn’t one. After the maintainer’s review, the wait also stops as soon as a short posting can already be read, and pages that return 4xx/5xx skip the wait entirely. 1,500 visible characters is just the fast way out, with 15 s as the cap. (The 0 → 20 was measured on an earlier version of the wait.)
Thanks for the correction — that's an important distinction, and good of you to flag your own numbers. The merged version sounds like the right shape: stop as soon as a short posting is readable, skip the wait entirely on 4xx/5xx, and let 1,500 characters just be the fast way out with 15 s as the cap. And noted that the 0 → 20 was measured on the earlier wait — the maintainer review clearly sharpened it.
This failure mode deserves its own name, because it shows up everywhere once you start looking: a system reporting completion for work it never actually did. The scanner wasn't lying exactly — "done" had just quietly come to mean "the loop reached this posting" instead of "the posting was read and processed."
The fix that generalizes: make the done-state impossible to reach without the evidence attached. If marking a posting done requires the extracted fields to exist — non-empty title, a real description span, a timestamp from the posting itself — then the state can't be faked by a code path that skipped the work. The claim and the proof become the same object.
Anywhere a status can be set by the same code that's supposed to do the work, with nothing downstream checking, you'll eventually get a green dashboard over an empty room.
Thanks, that’s close to how it ended up being fixed. #5814 (by bercedev) made the pipeline mark a posting done only when the evaluation returns a report. My #6169 replaced the fixed 2-second wait with a wait for the description text, so a page that can’t be read now stays pending instead of being marked done.
One difference from what you describe: the check is whether the page can be read at all, not field by field.
A question back: in agent pipelines, the evidence can be made up too. A model will happily fill in a title and description it never received. How do you make sure the evidence came from the source and not from the model?
You've named the failure mode exactly — if the model writes the evidence, the evidence is fiction with good formatting. Our rule: the model never originates a fact. The page is fetched and snapshotted first, with its content hash recorded at collection time, and every field in the output has to point back into that snapshot — the span it came from, not just the page it claims to be about. A separate deterministic check then re-reads the snapshot and compares: if a title or description can't be traced to actual text in it, the field comes out UNKNOWN, never filled in. The hash matters because it makes the snapshot itself the thing on trial later — anyone can re-fetch, re-hash, and see whether what was evaluated is what was reported. The model is only ever allowed to summarize what the snapshot proves is there.
dev.to/houssain_elmarouni_09942/th...
The stub that always answers "Score: 4.2/5" is a great way to prove the bug. It shows the tracker could not tell a real evaluation from an evaluation of nothing, which is the whole problem in one line.
I also like that the fix ended up as a gate rather than a log line. A count of characters received tells you about the damage afterwards. Refusing to mark a posting done until the page could actually be read stops it from happening, and a posting that stays pending is honest in a way a fake score never is.
We rebuilt a candidate pipeline at Ascendra Ventures around the same idea. Enrichment touched resumes, LinkedIn data, batch parsing and embeddings, and any slow step could leave a person in an unclear state. We moved it to a durable workflow where every person has to end in a final status, finished or visibly failed, with polling as a fallback when webhooks went quiet. Your fix and ours land on the same rule: nothing gets to look complete without evidence.
After #6169, do pages that never pass the threshold show up anywhere as a separate list, or do they just stay pending in the tracker?
Thanks — that’s the point: a pending row is honest, a 4.2/5 from nothing isn’t.
To your question: no separate list. A page that never becomes readable stays - [ ] in pipeline.md, the run prints why (No job description to evaluate at ()), and the next run tries it again. A posting the ATS reports as closed (Workday canApply: false, SmartRecruiters active: false) is handled the same way: not evaluated, left pending.