The line that stuck with me is 'accepted is not fetched' — two green checks that only ever proved my own egress is such a clean summary of the trap. Asserting on the artifact itself, parsing ftyp/moov/mvhd out of the file instead of trusting the echoed parameters, is the guard I'd steal first. With the hardcoded UNFETCHABLE_HOSTS list, do you ever re-probe a host to see if it recovers, or does a host stay on the blocklist once it fails?
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
Permanent until something in the worker's region fetches it, because a re-probe from my box is exactly the evidence that misled me in the first place. The unblock I'd accept is one job on that host with terminal_at - accepted_at under 60s; below that the mirror stays automatic and nobody has to remember the tuple.
That distinction between a re-probe from your box and a fetch from the worker's region is the crux of it — the mirror can only be trusted when the check runs where the failure lives. The 60-second bound on terminal_at minus accepted_at is a clean, checkable rule too; did that threshold come from the lag you observed in your incidents, or is it a round number you picked as a starting point?
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
Round number, and I'd rather say that than dress it up. I keep accepted_at, first_fetch_at and terminal_at as three separate fields, so after a week of good runs I can replace the 60s with the p95 of first_fetch_at - accepted_at instead of leaving my guess in the check.
That's a solid plan — letting a week of real runs set the number beats defending a guess, and keeping the three timestamps separate makes the p95 swap straightforward. Thanks for spelling out the reasoning behind it.
This is a good reminder that a 200 from my own box only proves my egress, not the worker's. I would add your 1x1 PNG probe as a regular check, logging accepted_at against terminal_at so a minutes-long dispatch gap shows up before users notice it.
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
I ran that probe twice before writing it up: 6m46s to terminal, both times, and the third number that matters is the reason, because it never changed. Log the triple (accepted_at, terminal_at, reason) and set the alert from your slowest known-good render, not from the queue's p99 — mine was 5m56s on a host the worker could reach, so a 10-minute threshold flags the broken run on the first try instead of the second.
That 10-minute threshold makes sense — basing it on your slowest known-good render, not the queue's p99, is the part I would have missed. Logging the reason alongside the two timestamps is a good call too, since a stable reason is what turns a slow run into a diagnosable one.
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
The reason field paid for itself on the first two runs: both failures ended on the same line, media URL could not be downloaded (timeout or source server error) at 6m46s each, while the good render off a domestic bucket finished in 5m56s. So the alert is two conditions — elapsed over 10m, or a terminal reason I haven't seen in the last 30 good runs — and it's the second one that fires on run #1 instead of run #3.
That second condition is the clever part — an unseen terminal reason fires on run one instead of waiting for three slow failures. With both failures landing at exactly 6m46s against a 5m56s good run, that rule looks solid.
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
One number behind it: the 30-run window is there because the render pool has two terminal reasons that only show up under load, so a window that short is what separates noise from a new failure mode. Cheapest version is a counter over the last 30 terminal reasons and an alert the first time any single reason goes 0 -> 1, no baseline math needed.
We're a place where coders share, stay up-to-date and grow their careers.
The line that stuck with me is 'accepted is not fetched' — two green checks that only ever proved my own egress is such a clean summary of the trap. Asserting on the artifact itself, parsing ftyp/moov/mvhd out of the file instead of trusting the echoed parameters, is the guard I'd steal first. With the hardcoded UNFETCHABLE_HOSTS list, do you ever re-probe a host to see if it recovers, or does a host stay on the blocklist once it fails?
Permanent until something in the worker's region fetches it, because a re-probe from my box is exactly the evidence that misled me in the first place. The unblock I'd accept is one job on that host with terminal_at - accepted_at under 60s; below that the mirror stays automatic and nobody has to remember the tuple.
That distinction between a re-probe from your box and a fetch from the worker's region is the crux of it — the mirror can only be trusted when the check runs where the failure lives. The 60-second bound on terminal_at minus accepted_at is a clean, checkable rule too; did that threshold come from the lag you observed in your incidents, or is it a round number you picked as a starting point?
Round number, and I'd rather say that than dress it up. I keep
accepted_at,first_fetch_atandterminal_atas three separate fields, so after a week of good runs I can replace the 60s with the p95 offirst_fetch_at - accepted_atinstead of leaving my guess in the check.That's a solid plan — letting a week of real runs set the number beats defending a guess, and keeping the three timestamps separate makes the p95 swap straightforward. Thanks for spelling out the reasoning behind it.
This is a good reminder that a 200 from my own box only proves my egress, not the worker's. I would add your 1x1 PNG probe as a regular check, logging accepted_at against terminal_at so a minutes-long dispatch gap shows up before users notice it.
I ran that probe twice before writing it up: 6m46s to terminal, both times, and the third number that matters is the reason, because it never changed. Log the triple (accepted_at, terminal_at, reason) and set the alert from your slowest known-good render, not from the queue's p99 — mine was 5m56s on a host the worker could reach, so a 10-minute threshold flags the broken run on the first try instead of the second.
That 10-minute threshold makes sense — basing it on your slowest known-good render, not the queue's p99, is the part I would have missed. Logging the reason alongside the two timestamps is a good call too, since a stable reason is what turns a slow run into a diagnosable one.
The reason field paid for itself on the first two runs: both failures ended on the same line,
media URL could not be downloaded (timeout or source server error)at 6m46s each, while the good render off a domestic bucket finished in 5m56s. So the alert is two conditions — elapsed over 10m, or a terminal reason I haven't seen in the last 30 good runs — and it's the second one that fires on run #1 instead of run #3.That second condition is the clever part — an unseen terminal reason fires on run one instead of waiting for three slow failures. With both failures landing at exactly 6m46s against a 5m56s good run, that rule looks solid.
One number behind it: the 30-run window is there because the render pool has two terminal reasons that only show up under load, so a window that short is what separates noise from a new failure mode. Cheapest version is a counter over the last 30 terminal reasons and an alert the first time any single reason goes 0 -> 1, no baseline math needed.