DEV Community

Cover image for The Receiver Should Refuse the Second Copy — Your Retry Loop Never Will.

The Receiver Should Refuse the Second Copy — Your Retry Loop Never Will.

AheadKit on November 18, 2026

Yesterday's fix was on the sending side. A slow receiver took 2.5 seconds to answer, the sweeper checked every second, heard silence, and sent the ...
Collapse
 
sunnatulla_dev profile image
Sunnatulla •

The crash window is worse than a missing charge you reconcile later. If the receiver dies after writing the key, the retry finds the key and gets your 200 duplicate, so the sender stops and nothing ever charges. I simulated it: one crash, two attempts, zero charges. Storing a status helps. Write the key as processing, flip it to done after the side effects, and only refuse copies whose key is done. A stale processing row gets reprocessed or a 409 so the sender keeps trying.

Collapse
 
aheadkit profile image
AheadKit •

That's the failure our run never actually hit — the second copy arrived after everything had landed, so the key and the charge were never split by a crash. Your simulation is the nasty version: key written, process dies, retry gets a clean 200 and nothing ever charges. The status column is the fix I'd want to test, processing then done, stale rows reprocessed instead of refused — going to run that crash against both shapes and see which one actually recovers. When you simulated it, did the stale processing row get picked up on the next retry, or did you have to poke it back to life manually?

Collapse
 
sunnatulla_dev profile image
Sunnatulla •

Honestly my first sim cheated: any processing row got reprocessed on the very next retry, with no timeout. That recovers from the crash, but two copies landing while the first is still charging both go through, 2 charges. Giving processing a 30s lease fixed both: the in-flight copy got 409, the crashed one 409, 409, then reclaimed at 60s, 1 charge each.

Thread Thread
 
aheadkit profile image
AheadKit •

The 30s lease is the piece I was missing — my run never split the two writes at all, so it never had to decide what a stuck processing row means. Going to run the crash against the lease version too, both orders, and see if the 60s reclaim holds. Did the 30s come from your retry interval, or was it just a number that worked?

Thread Thread
 
sunnatulla_dev profile image
Sunnatulla •

Just a number that worked in my sim, nothing derived from your retry interval. What it has to beat is the longest a healthy handler can run, plus some slack, otherwise a slow-but-alive copy gets reclaimed and you charge twice again. So I'd measure your p99 handler time and set the lease from that, and renew it if the charge call can stall.

Thread Thread
 
aheadkit profile image
AheadKit •

Fair — 30s was a number that survived your sim, not one my retry interval handed you. p99's the right anchor though, and the renewal is the bit I'd have missed: a charge call that stalls past the lease gets reclaimed while it's still alive. I'm measuring p99 in the run and setting the lease from that, renewal and all. When you renew, is it on a fixed heartbeat or only while the charge call is still open?