DEV Community

Cover image for The Receiver Should Refuse the Second Copy — Your Retry Loop Never Will.
AheadKit
AheadKit

Posted on

The Receiver Should Refuse the Second Copy — Your Retry Loop Never Will.

Yesterday's fix was on the sending side. A slow receiver took 2.5 seconds to answer, the sweeper checked every second, heard silence, and sent the same order three times. The fix was a claim row: one row per delivery key, claimed once, so the second and third ticks had nothing to send. Three deliveries became one.

Then itsvicmoreno asked the question that undoes that fix, politely, in the comments. What if the acknowledgement is lost AFTER the write? The receiver has already taken the order, already charged the card, already sent the email. The sender never hears back, decides the delivery failed, and sends it again. The claim row cannot help you there. The sender did its job, the row says done or it says nothing useful, and a second copy is already on its way. Does the fix live only in the sweeper, or does the receiving end have a say?

I ran that exact case on my own machine. One event, evt-1042, a $29.00 payment confirmation for order ORD-1042. The same event delivered twice, the second send waiting for the first reply, so the second copy arrives after the first write has fully landed. Two receivers, side by side, with the same side effects: a charge line, a confirmation email line, and one stock decrement each time a delivery is accepted. The only difference is that one of them keeps a table of keys it has already processed, and checks it before doing anything else.

My claim is simple, and I expect some of you to disagree with it. A smarter retry loop will never be enough. The receiver has to be able to say no to the second copy, because the receiver is the last place where a duplicate can still be stopped before it becomes a second charge.

Problem 1: the plain inbox charged twice for one order

The plain receiver has no duplicate check at all. Every delivery it accepts is a real delivery, as far as it knows. Here is what two copies of evt-1042 did to it:

Plain inbox Receiver with a processed-keys table
Deliveries received 2 2 (1 accepted, 1 refused)
Charges 2, $29.00 twice, $58.00 in total 1, $29.00
Confirmation emails 2 1
Stock decrements 2, stock 10 down to 8 1, stock 10 down to 9
Second copy's reply 200, processed 200, duplicate, refused

Both runs gave the same numbers, twice each, at 00:07:04 and 00:07:08 for the plain receiver, 00:07:06 and 00:07:11 for the one with the table. The plain receiver logged two deliveries for one order, charged $58.00 for a $29.00 order, sent the customer the same confirmation twice, and took two items out of stock for one sale. Nothing about the second copy looked wrong when it arrived. It had the same key, the same order, the same amount, and it arrived after a perfectly normal gap. A receiver that cannot recognise a key it has already seen has no way to know it has been paid twice.

That is the shape of the bug itsvicmoreno pointed at. The sender-side claim is a good fix for the slow case, and I am not taking that back. But the lost-acknowledgement case walks straight past it. The sender believes the first delivery failed. From the sender's side, resending is the correct, careful thing to do. The duplicate is not a mistake in the retry logic. It is the retry logic working as designed, on wrong information.

Problem 2: the table works, and the order of the two steps is the whole trick

The second receiver keeps a small table, one row per delivery key. When a delivery arrives, it checks the table first. If the key is already there, it refuses the copy before any side effect runs, and still answers 200, so the sender stops retrying. If the key is new, it writes the key into the table first, and only then charges, emails, and decrements stock.

The reply matters as much as the refusal. The second copy got {"status":"duplicate","refused":true} with a 200 behind it. A refused copy that answers with an error would look like another failed delivery, and the sender would send it a third time, then a fourth. A refusal has to sound like success to the sender, because from the sender's point of view the order did go through. It went through the first time.

And the order of the two steps inside the receiver is the whole trick. Key first, then side effects. If you charge first and write the key second, there is a gap between the two, and a crash in that gap leaves a charge with no record of it. The next copy arrives, the table says it is new, and the customer is charged again. Writing the key first closes that gap in one direction: a crash after the write leaves a recorded key with no charge, which is a missing charge you can reconcile, not a double charge you have to refund. One of those is a bookkeeping problem. The other is a customer with two charges on their statement.

The check itself is small. In the receiver, before anything else happens:

if (keys.includes(evt.delivery_key)) {
  // refuse: already processed, no side effects
  res.end(JSON.stringify({ status: 'duplicate', refused: true }));
  return;
}
// record the key FIRST, then charge, email, decrement stock
writeKeys([...keys, evt.delivery_key]);
Enter fullscreen mode Exit fullscreen mode

That is the entire defence. A lookup, a refusal, and a write that happens before the money moves.

Problem 3: the table does not remove the duplicate, only its cost

Be honest about what the table buys you. The second copy still arrives. It still costs a delivery, a lookup, and a reply. On a bad day, when every acknowledgement is being lost, the table does not reduce the traffic at all. It reduces the damage the traffic can do. Two copies still cross the wire. Only one of them touches money, email, or stock.

That is a narrower win than the headline version, and it is worth saying plainly, because it decides where this fix belongs. The table is not a replacement for the sender-side claim. It is the second layer, the one that catches the copy the claim could not stop. The claim stops the sweeper resending a delivery that is still in flight. The table stops the receiver acting on a delivery that already landed. You need both, because they guard different moments, and the lost-acknowledgement case is the one that slips between them.

Problem 4: the honest limits of this test

Three limits, because the test was small. First, my two copies arrived one after the other, in order, never at the same instant. Two copies landing at the exact same moment, both checking the table before either has written, were not tested, and a race like that is exactly where a simple lookup breaks. A real table wants the check and the write to happen as one step, not two, and I did not test that step.

Second, the crash window I described above, between writing the key and running the side effects, was not tested either. I know the order matters. I did not prove what happens when the receiver dies inside it. A real table wants the key write and the side effects in one transaction, or a reconciliation pass that finds recorded keys with no charge. My test had neither.

Third, I reset the table before each run. Whether the keys survive a receiver restart, and for how long they should be kept before they are cleared, are both open questions my test does not answer. A table that forgets is a table that charges twice, just later.

So the claim stands, narrowed to what the numbers support. The sender-side claim fixed the slow receiver. It does not fix the lost acknowledgement, because in that case the sender is right to resend. The receiver-side table does fix it: the second evt-1042 was refused, one charge instead of two, one email instead of two, stock 9 instead of 8. The fix belongs at the receiving end, in a table checked before the money moves, with the key written before the side effects run.

If you run webhooks, you have met this one. What did your duplicate cost you, a double charge, a double email, a customer asking why they got two copies of the same thing? And what caught it in the end, a table like this one, a claim on the sending side, or something else entirely?


P.S. Everything I write stays free — that part doesn't change. But if this one was worth something to you, you can buy me a coffee ☕ (one-time or monthly, both work). It covers the API credits and hosting behind these experiments — and honestly, it's also the nudge that gets the next one started on the evenings I'd otherwise skip it. — Salam

Buy me a coffee

AheadKit on Gumroad — Beginner-Friendly Books & Packs

Top comments (12)

Collapse
 
sunnatulla_dev profile image
Sunnatulla •

The crash window is worse than a missing charge you reconcile later. If the receiver dies after writing the key, the retry finds the key and gets your 200 duplicate, so the sender stops and nothing ever charges. I simulated it: one crash, two attempts, zero charges. Storing a status helps. Write the key as processing, flip it to done after the side effects, and only refuse copies whose key is done. A stale processing row gets reprocessed or a 409 so the sender keeps trying.

Collapse
 
aheadkit profile image
AheadKit •

That's the failure our run never actually hit — the second copy arrived after everything had landed, so the key and the charge were never split by a crash. Your simulation is the nasty version: key written, process dies, retry gets a clean 200 and nothing ever charges. The status column is the fix I'd want to test, processing then done, stale rows reprocessed instead of refused — going to run that crash against both shapes and see which one actually recovers. When you simulated it, did the stale processing row get picked up on the next retry, or did you have to poke it back to life manually?

Collapse
 
sunnatulla_dev profile image
Sunnatulla •

Honestly my first sim cheated: any processing row got reprocessed on the very next retry, with no timeout. That recovers from the crash, but two copies landing while the first is still charging both go through, 2 charges. Giving processing a 30s lease fixed both: the in-flight copy got 409, the crashed one 409, 409, then reclaimed at 60s, 1 charge each.

Thread Thread
 
aheadkit profile image
AheadKit •

The 30s lease is the piece I was missing — my run never split the two writes at all, so it never had to decide what a stuck processing row means. Going to run the crash against the lease version too, both orders, and see if the 60s reclaim holds. Did the 30s come from your retry interval, or was it just a number that worked?

Thread Thread
 
sunnatulla_dev profile image
Sunnatulla •

Just a number that worked in my sim, nothing derived from your retry interval. What it has to beat is the longest a healthy handler can run, plus some slack, otherwise a slow-but-alive copy gets reclaimed and you charge twice again. So I'd measure your p99 handler time and set the lease from that, and renew it if the charge call can stall.

Thread Thread
 
aheadkit profile image
AheadKit •

Fair — 30s was a number that survived your sim, not one my retry interval handed you. p99's the right anchor though, and the renewal is the bit I'd have missed: a charge call that stalls past the lease gets reclaimed while it's still alive. I'm measuring p99 in the run and setting the lease from that, renewal and all. When you renew, is it on a fixed heartbeat or only while the charge call is still open?

Thread Thread
 
sunnatulla_dev profile image
Sunnatulla •

Heartbeat, but only from inside the open call, every lease/3 or so, and the renew is conditional on the owner token still being mine. That still breaks on a frozen process, because its beats stop too. I simulated A blocked for 40s with a 30s lease. B reclaims and charges, A wakes, its renew and finish both get refused, but A's charge call still went out. Only passing the event id as the provider's idempotency key made that one charge instead of two.

Thread Thread
 
aheadkit profile image
AheadKit •

The frozen-process case is the one my run never hit — my crashes split the key and the charge, yours lets the charge call go out after the lease is already gone. Going to run that shape, A frozen mid-charge with B already in. One thing I want to get right in it: was A's charge call part of the finish attempt after it woke, or was it already in flight when it froze?

Thread Thread
 
sunnatulla_dev profile image
Sunnatulla •

Neither, quite. In mine A froze inside the charge step but before the request left, so it went out after A woke, and nothing between waking and sending looked at the failed renew. If you gate the send on a fresh conditional renew, that case drops to one charge. Freeze A right after that check passes and it's two again, and the same if the request was already on the wire. That gap is why I ended up trusting only the provider key.

Thread Thread
 
aheadkit profile image
AheadKit •

That gate on a fresh conditional renew is the piece I was missing — it turns your exact case into one charge. Freezing right after the check passes is the hole that stays, and that's the run: freeze at each of those three points and count the charges. Did you ever try the gate in your sim, or did the provider key make it not worth bothering with?

Thread Thread
 
sunnatulla_dev profile image
Sunnatulla •

Only in a toy model, not against a real provider. Ran your three points just now: freeze before the renew gives 1 charge, freeze after the renew passed gives 2, freeze with the request already in flight gives 2. With a provider idempotency key all three come out at 1. So for me the gate only closes the first case and the key is what actually saves you. I'd keep the gate anyway since it's cheap and it stops most of the doubled work, just not the money part.

Thread Thread
 
aheadkit profile image
AheadKit •

That's the clean verdict — the gate closes the first case and the key does the money work. Keeping the gate anyway is the right call, it's free and it stops the doubled work even when the money's already safe. Was it the in-flight freeze that made you trust the key over the gate, or the after-renew one?