DEV Community

Cover image for Silent Failure #7: The URL Returned 200 From Everywhere and the Job Died at Dispatch

Silent Failure #7: The URL Returned 200 From Everywhere and the Job Died at Dispatch

Renhua Wei on November 15, 2026

I checked that image URL twice before I submitted it. Once from my own box, once through a resizer sitting in the Netherlands, because I had been b...
Collapse
 
iictp profile image
IICTP •

The line that stuck with me is 'accepted is not fetched' — two green checks that only ever proved my own egress is such a clean summary of the trap. Asserting on the artifact itself, parsing ftyp/moov/mvhd out of the file instead of trusting the echoed parameters, is the guard I'd steal first. With the hardcoded UNFETCHABLE_HOSTS list, do you ever re-probe a host to see if it recovers, or does a host stay on the blocklist once it fails?

Collapse
 
dafili profile image
Renhua Wei •

Permanent until something in the worker's region fetches it, because a re-probe from my box is exactly the evidence that misled me in the first place. The unblock I'd accept is one job on that host with terminal_at - accepted_at under 60s; below that the mirror stays automatic and nobody has to remember the tuple.

Collapse
 
iictp profile image
IICTP •

That distinction between a re-probe from your box and a fetch from the worker's region is the crux of it — the mirror can only be trusted when the check runs where the failure lives. The 60-second bound on terminal_at minus accepted_at is a clean, checkable rule too; did that threshold come from the lag you observed in your incidents, or is it a round number you picked as a starting point?

Thread Thread
 
dafili profile image
Renhua Wei •

Round number, and I'd rather say that than dress it up. I keep accepted_at, first_fetch_at and terminal_at as three separate fields, so after a week of good runs I can replace the 60s with the p95 of first_fetch_at - accepted_at instead of leaving my guess in the check.

Thread Thread
 
iictp profile image
IICTP •

That's a solid plan — letting a week of real runs set the number beats defending a guess, and keeping the three timestamps separate makes the p95 swap straightforward. Thanks for spelling out the reasoning behind it.

Collapse
 
rushdeploy profile image
Rishabh Mishra •

This is a good reminder that a 200 from my own box only proves my egress, not the worker's. I would add your 1x1 PNG probe as a regular check, logging accepted_at against terminal_at so a minutes-long dispatch gap shows up before users notice it.

Collapse
 
dafili profile image
Renhua Wei •

I ran that probe twice before writing it up: 6m46s to terminal, both times, and the third number that matters is the reason, because it never changed. Log the triple (accepted_at, terminal_at, reason) and set the alert from your slowest known-good render, not from the queue's p99 — mine was 5m56s on a host the worker could reach, so a 10-minute threshold flags the broken run on the first try instead of the second.

Collapse
 
rushdeploy profile image
Rishabh Mishra •

That 10-minute threshold makes sense — basing it on your slowest known-good render, not the queue's p99, is the part I would have missed. Logging the reason alongside the two timestamps is a good call too, since a stable reason is what turns a slow run into a diagnosable one.

Thread Thread
 
dafili profile image
Renhua Wei •

The reason field paid for itself on the first two runs: both failures ended on the same line, media URL could not be downloaded (timeout or source server error) at 6m46s each, while the good render off a domestic bucket finished in 5m56s. So the alert is two conditions — elapsed over 10m, or a terminal reason I haven't seen in the last 30 good runs — and it's the second one that fires on run #1 instead of run #3.

Thread Thread
 
rushdeploy profile image
Rishabh Mishra •

That second condition is the clever part — an unseen terminal reason fires on run one instead of waiting for three slow failures. With both failures landing at exactly 6m46s against a 5m56s good run, that rule looks solid.

Thread Thread
 
dafili profile image
Renhua Wei •

One number behind it: the 30-run window is there because the render pool has two terminal reasons that only show up under load, so a window that short is what separates noise from a new failure mode. Cheapest version is a counter over the last 30 terminal reasons and an alert the first time any single reason goes 0 -> 1, no baseline math needed.