career-ops is an open-source job-search tool with over 80,000 stars on GitHub. One part of it, openrouter-runner, takes the job postings in your pipeline, opens each page in a headless browser, sends the job text to a free model on OpenRouter, and writes a score into your tracker.
On 23 October I ran it on 15 live Workday postings (career-ops v1.37.0). All 15 rows were marked done. All 15 got a report.
Then I counted what the model had actually received.
What the model saw
Between 0 and 397 characters of job text per posting. Six of the 15 got nothing at all. The rest got the page frame, for example:
Skip to main content Loading Follow Us © 2026 Workday, Inc. All rights reserved.
That is 80 characters. No title, no requirements, no description.
The cause was one line. After opening the page, the runner waited a fixed 2 seconds, then read the body. Workday builds the job description with JavaScript after the page loads. When I reran the same postings the next day, the description appeared 5.0 to 9.2 seconds after navigation started, on 13 of the 15. Two seconds was never going to be enough.
Why "done" was the real bug
A slow page is normal. The bug was what happened next: the pipeline treated "the model replied" as "the posting was evaluated".
To show it without any real model, I pointed the runner at three local pages with no job text, and replaced the model with a stub that always answers Score: 4.2/5. All three went out with 0 characters of job text. All three were marked done. The tracker said 4.2/5 | Evaluated.
Nothing in the tracker separates a real score from a score about nothing. Someone reading it would apply, or skip, based on a number that came from an empty page.
A second bug in the same runner
The runner reuses the prompt that career-ops gives to agents that have tools. That prompt tells the model to search the web, run other scripts, check the CV. The runner sends no tools.
So the free models did what the prompt asked. On Workday pages with no description, 19 of 19 replies were tool calls, or said they were about to make one. On real descriptions (7.8K to 16K characters), 4 replies came back: 3 tool calls and 1 empty. Across all 23 replies, the score parser found 0 scores.
How it got fixed
I filed both bugs as issue #5809 on 24 October, with the numbers above and the stub script. Fixing it took four pull requests. Three are mine:
| PR | What changed | Merged |
|---|---|---|
| #5814 (by bercedev) | A posting that wasn't read is not saved or marked done | 25 Oct |
| #5817 | The runner tells the model it has no tools, and strips the steps it can't run | 26 Oct |
| #6017 | Model timeout raised from 15 s to 120 s | 27 Oct |
| #6169 | Wait for the job description, not for a fixed 2 s | 6 Nov |
6017 came out of the same work. The 15-second timeout was set in June, when the two prompt files the runner sends added up to about 31K characters. By October they were about 113K. On the free model rotation, a timeout also put the model on a blacklist that survives restarts.
After #5814, the false "done" was gone. But it showed the next problem: with the 2-second wait still there, a Workday posting could never be read, so its row stayed pending forever. I measured 20 live Workday postings, 4 each from NVIDIA, Salesforce, Adobe, Intel and Mastercard:
- With the fixed 2 s wait: 0 of 20 readable. All refused as too short, in 3 to 4 seconds.
- With a wait for the job text: 20 of 20 readable, 3,680 to 9,072 characters, in 3.3 to 6.9 seconds.
6169 first tries career-ops' own job-board API reader (Workday, Greenhouse, Lever and others), and only falls back to the browser if that returns nothing. In the browser it waits until the main content holds more than 1,500 visible characters, or until the page clearly says the posting is closed, with a 15-second cap. A 4xx or 5xx page skips the wait.
What review added
The maintainer asked for changes, and the automated reviewer (CodeRabbit) raised findings over several rounds. Not all of it was noise. One finding was real, and it came from my own change: a page with an "Apply now" button and a description still saying "Loading..." could end the wait early. I wrote a test for exactly that page, watched it fail, then fixed it.
6169 was merged on 6 November and shipped in career-ops v1.40.0 on 10 November. On the issue, the maintainer wrote: "Thanks for fixing it yourself."
What I check now, in any pipeline that sends fetched content to a model
- Count what the model received. Characters, not "page loaded". It is one log line.
- Don't mark work done because a reply came back. Mark it done when the input passed a check.
- Don't let the prompt promise tools the code doesn't send. The model will try to use them.
- Wait for content, not for a clock. A fixed sleep is a guess about someone else's server.
The tests passed the whole time, and the tracker looked full. That is what made it hard to see.
Links
- Issue: https://github.com/career-ops-hq/career-ops/issues/5809
- PRs: #5814, #5817, #6017, #6169
- Release notes: https://github.com/career-ops-hq/career-ops/blob/main/CHANGELOG.md
AI disclosure: I found and fixed these bugs with the help of AI coding tools (Claude Code), and drafted this post with Claude. Every number comes from the linked issue and pull requests, and I checked the text before publishing.
Top comments (11)
The "count what the model received" rule is the one I'm stealing — a single log line of characters received would have caught this on day one. A tracker showing a 4.2/5 score on zero characters of job text is the perfect illustration of why "the model replied" must never equal "the work is done". Great write-up, and the four PRs make the fix easy to follow.
Thanks, glad it’s useful. One thing I’d add: a log line only tells you after the damage, so the fix turns the count into a gate. #5814 stops a posting that wasn’t read from being saved or marked done, and #6169 waits until the main content passes 1,500 visible characters (15 s cap) instead of a fixed 2 s. On 20 live Workday postings, that took readable pages from 0 to 20.
Fair point — the log line is the autopsy, the gate is the prevention. 0 to 20 readable pages on live Workday postings is a clean before/after, and the 1,500-character threshold with a 15 s cap is a nice balance: strict enough to mean something without timing out on slow pages.
Agreed. One correction to my own numbers, because 1,500 sounds like a hard minimum and in the merged version it isn’t one. After the maintainer’s review, the wait also stops as soon as a short posting can already be read, and pages that return 4xx/5xx skip the wait entirely. 1,500 visible characters is just the fast way out, with 15 s as the cap. (The 0 → 20 was measured on an earlier version of the wait.)
Thanks for the correction — that's an important distinction, and good of you to flag your own numbers. The merged version sounds like the right shape: stop as soon as a short posting is readable, skip the wait entirely on 4xx/5xx, and let 1,500 characters just be the fast way out with 15 s as the cap. And noted that the 0 → 20 was measured on the earlier wait — the maintainer review clearly sharpened it.
This failure mode deserves its own name, because it shows up everywhere once you start looking: a system reporting completion for work it never actually did. The scanner wasn't lying exactly — "done" had just quietly come to mean "the loop reached this posting" instead of "the posting was read and processed."
The fix that generalizes: make the done-state impossible to reach without the evidence attached. If marking a posting done requires the extracted fields to exist — non-empty title, a real description span, a timestamp from the posting itself — then the state can't be faked by a code path that skipped the work. The claim and the proof become the same object.
Anywhere a status can be set by the same code that's supposed to do the work, with nothing downstream checking, you'll eventually get a green dashboard over an empty room.
Thanks, that’s close to how it ended up being fixed. #5814 (by bercedev) made the pipeline mark a posting done only when the evaluation returns a report. My #6169 replaced the fixed 2-second wait with a wait for the description text, so a page that can’t be read now stays pending instead of being marked done.
One difference from what you describe: the check is whether the page can be read at all, not field by field.
A question back: in agent pipelines, the evidence can be made up too. A model will happily fill in a title and description it never received. How do you make sure the evidence came from the source and not from the model?
You've named the failure mode exactly — if the model writes the evidence, the evidence is fiction with good formatting. Our rule: the model never originates a fact. The page is fetched and snapshotted first, with its content hash recorded at collection time, and every field in the output has to point back into that snapshot — the span it came from, not just the page it claims to be about. A separate deterministic check then re-reads the snapshot and compares: if a title or description can't be traced to actual text in it, the field comes out UNKNOWN, never filled in. The hash matters because it makes the snapshot itself the thing on trial later — anyone can re-fetch, re-hash, and see whether what was evaluated is what was reported. The model is only ever allowed to summarize what the snapshot proves is there.
dev.to/houssain_elmarouni_09942/th...
The stub that always answers "Score: 4.2/5" is a great way to prove the bug. It shows the tracker could not tell a real evaluation from an evaluation of nothing, which is the whole problem in one line.
I also like that the fix ended up as a gate rather than a log line. A count of characters received tells you about the damage afterwards. Refusing to mark a posting done until the page could actually be read stops it from happening, and a posting that stays pending is honest in a way a fake score never is.
We rebuilt a candidate pipeline at Ascendra Ventures around the same idea. Enrichment touched resumes, LinkedIn data, batch parsing and embeddings, and any slow step could leave a person in an unclear state. We moved it to a durable workflow where every person has to end in a final status, finished or visibly failed, with polling as a fallback when webhooks went quiet. Your fix and ours land on the same rule: nothing gets to look complete without evidence.
After #6169, do pages that never pass the threshold show up anywhere as a separate list, or do they just stay pending in the tracker?
Thanks — that’s the point: a pending row is honest, a 4.2/5 from nothing isn’t.
To your question: no separate list. A page that never becomes readable stays - [ ] in pipeline.md, the run prints why (No job description to evaluate at ()), and the next run tries it again. A posting the ATS reports as closed (Workday canApply: false, SmartRecruiters active: false) is handled the same way: not evaluated, left pending.