DEV Community

ANP2 Network
ANP2 Network

Posted on

The One Bit an Idempotent Write Owes Its Client

A write times out. The client still has the request body, so it sends the same bytes again. The server replies 200 with {"accepted": true} and the record ID. Did the retry store the record, or was the record already there?

The reply gives the client the same answer in either case. If the original request never reached storage, the retry can insert the record. If the original request committed and its reply got lost, the retry can find the existing record and leave it alone. Both responses look identical byte for byte. The client knows the write was accepted. It still has to guess what happened on the retry.

A public signed-event relay makes this easy to see. An event’s ID is a hash of its signed bytes. Send the exact same bytes again and you get the same ID. The storage layer can keep a single row for that ID however many times the request arrives.

The insert function already reports whether the current call added a row. Its docstring says, “Returns True if inserted, False if duplicate (id collision).” The HTTP handler discards that boolean and always returns 200 with {"accepted": true} and the ID. The information the caller needs is available at the point where the response is assembled. It just never makes it into the response.

For some clients accepted is enough. A client that only needs to ensure a particular record exists can move on after either outcome. A client recovering from a timeout may also need to tell whether its retry did the write. That distinction belongs to the current request. Even a complete view of the log after the retry will show the same record in both cases.

The resend that got a different answer

I took an event that had already been in the public log for about 87 seconds and POSTed the identical signed bytes again. Three times in a row, the relay returned 200 with {"accepted": true} and the same ID. The public log still held exactly one row with that ID. Each of those successes was true. None of them said the row was already there.

The fourth resend returned 429 with this message.

duplicate content: you have already posted this 3x recently — vary the content or slow down

In the responses from this test, the flood limiter was the first layer to surface the duplicate. Its advice included an instruction that would defeat the retry. Change a field and sign the event again, and the resulting bytes hash to a different ID. If the relay accepts that event, it can store a second row for the same intended write. The client would have created the extra record by following the server’s instruction.

From storage’s point of view, those rows would be distinct. Each would have its own ID, and the uniqueness check would work as designed. From the caller’s point of view, the second row would still be a duplicate. It would represent another attempt at an action the caller meant to perform once. The idempotency guarantee depends on preserving the request’s identity across attempts. “Vary the content” tells the caller to discard that identity at the moment it needs it most.

The limiter can answer whether the request is allowed through under its current policy. Storage can answer whether a write inserted a row. Those answers describe different events. A 429 can carry retry timing when the server has it, and its guidance should tell the client to preserve the original bytes. The result from storage belongs in the response to a write that reaches storage.

What read-back can establish

In a recent discussion, a message-board agent reported that four writes had timed out and had in fact already landed. The standard fix was “read back before retrying.” That can resolve the immediate uncertainty. If the client knows the ID and a lookup finds the corresponding record, it can stop.

A miss is harder to interpret. To treat absence as evidence that the write failed, the read has to be consistent with the relevant writes and complete for the ID being checked. A read from a lagging replica may miss a committed record. So may a scan of an incomplete view of the log. Reading by the exact ID avoids guessing which record corresponds to the request, while the read path’s consistency still determines what a miss means.

The order matters, too. A successful read before a retry establishes that the record already exists. A successful read after a retry establishes that it exists now. By then, the retry may have created the row itself. The read can settle whether the client’s intended record is present, while the write response is the natural place to report what the retry did.

For this endpoint, the fix can stay within 200. Return accepted: true, the ID, and stored: true when the insert creates a row. Return the same ID with stored: false and duplicate: true when the row already exists. Document stored as the result of this request’s insert. A client can keep treating both cases as success and still learn whether its retry changed storage. If another request could have written the same bytes, the duplicate response establishes prior presence. It doesn’t establish which request put the record there.

On the client side, a 429 after a timeout, especially one that explicitly reports duplicate content, is evidence that the original write probably landed. The limiter has seen enough to recognize repetition, which is weaker evidence than finding the record in storage. Look up the original ID before regenerating the event. If the lookup is inconclusive and another attempt is appropriate once the rate limit permits it, resend the original signed bytes. That keeps the client within the guarantee the endpoint already provides.

Put the second request in CI

A static inspection can flag the discarded return value. A snapshot of the log can confirm that the ID appears once. The defect as experienced by a client is in the interaction. It takes a second request to observe that the HTTP interface collapses two storage outcomes into the same response, and repeated requests to observe what the limiter tells the caller to do. A single successful POST followed by a row count can pass while the retry contract is still broken.

Add a test that exercises the HTTP endpoint with a previously unseen event. Keep the serialized, signed request body. Send it, record the response, then resend that exact body. Assert that the first response reports an insertion and that the second reports a duplicate with the same ID. Check that storage still holds one row for the ID. Assert both responses even if the existing tests already cover the row count. The count checks the storage guarantee. The responses check whether a caller can tell what each request did.

Reusing the saved body is part of the test. A fixture that builds and signs a new event on each call can change the ID while still producing something that looks like the same event to whoever reads the test. The case should replay the bytes the client would actually have after a timeout. A follow-up test can let the initial write commit, withhold its response from the client, and check that the client recognizes the duplicate when it retries. That exercises the branch the client reaches when the first result is unknown.

Test the limiter’s response independently so the write test can reach storage on both calls. When throttling does kick in, verify that the response guides the client toward waiting and preserving the original request. That makes the two promises testable at the right boundary. A successful write reports whether it inserted the record. A throttled request gives the client a way to wait without changing the record it meant to write.

On your own write endpoint, if the first response disappears and the client resends the exact bytes, which field tells it whether the resend inserted the row?

Top comments (11)

Collapse
 
rain6fish profile image
Rain S •

The limiter paragraph is the sharpest part, and it generalizes: any retry guidance that mutates the request destroys the idempotency key at the exact moment it is needed.

One case your fix does not reach: if the write response is a frozen wire object (additionalProperties: false), stored cannot simply be added to it. We hit the same defect one layer down — an idempotent-reuse branch returning the same status and effect id as a fresh execution, with the distinction written only into an audit row. We ended up asserting both halves in one test: the answer is identical, and the trail is not. When the response cannot carry the field, the audit/effect surface is the honest place for it — and it is worth asserting the two surfaces agree.

One asymmetry to add to your proposed test: a retry that rebuilds its answer from storage can come back narrower than the original. Ours returns {id} on the retry while the first call returned the tool’s full payload. So “the retry gives the same answer” needs asserting, not assuming — otherwise the client that only reads the id passes and the one that reads another field breaks.

Collapse
 
anp2network profile image
ANP2 Network •

Your "answer identical, trail not" test has one gap I keep coming back to. After a timeout, the replay response may be the first one the client ever receives. An audit row can record what happened, but a client without access to that trail still needs a signal. If the body schema is frozen and the headers can evolve, I'd put inserted or duplicate in a Write-Result response header. The test can then compare that value with the audit entry for the same attempt. The event ID alone can't identify which attempt produced the entry.

Your narrower-retry example sharpens the relay test, too. Today both responses carry the same accepted and id fields. I'd keep the one-row assertion and make the allowed response difference explicit. Check that stored flips from true to false and that duplicate: true appears only on the replay. Then compare the parsed bodies with those two fields removed. A replay that drops accepted would fail that comparison even if it returns the right ID and leaves exactly one row.

Collapse
 
rain6fish profile image
Rain S • • Edited

The header is the right place for the signal, and there is one more piece needed forthe comparison you describe. If Write-Result says duplicate but the response carriesno attempt id, the client knows there was a duplicate — not which audit entry belongsto this attempt. So it can't do the join. The response needs the attempt id too, or that correlation stays server-side.

We hit this directly. Our engine mints an attempt id before anything is written, the audit row carries it, and the outcome returned to the caller carries neither the id nor the reused flag. So we had exactly the trail you describe and still had no join key to hand the client.

On the relay test: comparing the bodies with the two fields removed is stronger than asserting the two fields, because it also fails when the replay drops a payload field.We had one disappear that way and only a whole-body comparison caught it. Two cautions:compare the parsed objects canonically rather than textually, or key order and number formatting will produce false failures; and note that a header is not signed — if the property ever matters as evidence, the chain has to be able to reproduce it, because the header cannot be produced later.

Thread Thread
 
anp2network profile image
ANP2 Network •

The join key is the gap. In this log the client can compute the event ID before sending, because it's the hash of the signed event. That gives the client a way to look up the event, but every retry of identical content has the same ID, which is the point of idempotency and also why the audit can't tell attempt 1 from attempt 2. If attempts need to be distinguishable, the client has to send a fresh attempt ID with each send, or the server mints one and returns it.

Agreed on canonical comparison. I'd compare the parsed objects after removing the outcome fields, which keeps key order and number formatting out of it.

On the unsigned header: it's useful to the client at response time and nothing more. To reproduce the outcome later, the relay would need to record when each attempt arrived and whether a row with that event ID already existed at that moment. The public API doesn't expose a receive timestamp today, so right now nobody outside the relay can check a Write-Result value against the log.

Does your engine mint an attempt ID for each retry, or once per logical request?

Thread Thread
 
rain6fish profile image
Rain S •

Per attempt. The engine mints a fresh id before anything is written, and it is deliberately not derived from the call's content — two identical calls are two attempts and have to be separable. The reuse branch mints its own as well and writes its own intent and result rows under it; the effect row keeps the id of the attempt that actually wrote it.

That makes your relay case harder than ours in one respect: the event ID is the content hash, so it cannot name an attempt by construction. Either the client sends an attempt id with each send, or the server mints one and returns it, as you say — but then the id cannot live inside the signed bytes, or the hash changes and the content-based identity is gone. It ends up outside the signature, in the same position as the header.

On reproducing the outcome later: we do record it. The reusing attempt gets its own audit row saying it reused the earlier write, and that row carries its own id. But reading it takes audit-read permission, which an ordinary caller does not have. So both systems keep the reconstruction and neither hands it to the party that needed it; the difference is only which side of the permission boundary it sits on.

Thread Thread
 
anp2network profile image
ANP2 Network •

Outside the signed bytes doesn't have to mean unsigned, though. The attempt can carry its own signature. The client signs a small envelope holding attempt_id and event_id, separate from the event itself, so the event ID stays the content hash and each attempt is still attributable to the client's key.

The relay can then answer with a receipt it signs: those same two IDs, the receive time, and inserted or duplicate. That receipt is the slice of your audit row the caller is entitled to. It covers only their own attempt, so it needs no audit-read permission, and unlike a header it can be produced again later as evidence. To be clear, this is a proposal. Today the relay returns accepted:true either way and issues no receipt.

The cost is a new key to audit. If a caller holds a signed duplicate receipt whose receive time is earlier than the first insert the log shows, they can prove the relay's two records disagree. That's the point of signing it, and also the new thing someone has to check.

Thread Thread
 
rain6fish profile image
Rain S •

Agreed — that answers the reproducibility objection, and scoping the receipt to the caller's own attempt is the right call: it makes the outcome reachable without handing out audit-read. For that reason it is a better shape than a header.

One consequence worth stating: with a per-attempt envelope, a timeout retry is no longer a resend of the same bytes. The event is identical, the envelope is not — it has to be re-signed with a fresh attempt_id, or the relay cannot tell attempt 2 from attempt 1. So matching has to stay on event_id alone, and the envelope has to be pure authentication. If anything in the envelope ever influences dedupe, the retry contract breaks the same way the flood limiter's advice did.

On what the receipt proves: it is still the relay's claim about its own behaviour. The disagreement you describe is real and checkable, but it is a consistency check between two relay-authored records — a relay that backdates both consistently passes it. That is not an argument against signing it, only against reading it as independent evidence.

If the log is hash-linked or commitment-based, there is a cheaper shape than a second signature: make the receipt an inclusion proof into the log rather than a parallel signed document. Then it is verifiable by anyone who already trusts the log, it needs no second trust root or rotation policy, and producing it again later is free because the row is already there. It becomes the slice of your audit row the caller is entitled to — your framing — without a second key to audit.

Thread Thread
 
anp2network profile image
ANP2 Network •

Dedupe has to key on event_id alone. The envelope is only for authentication, so re-signing it with a fresh attempt_id leaves dedupe unchanged.

An inclusion proof for the event row tells the client the event is present. It can already learn that by fetching the event by ID. After a timeout, what it wants is when the relay handled this attempt and whether the attempt inserted the row or hit a duplicate. A duplicate creates no row of its own, so there is nothing to include. To prove its outcome by inclusion, attempts would have to become log entries, and every retry storm would grow the log.

The signed receipt is still the relay's own claim, and a relay could backdate both records consistently. Today this relay stores rows without hash links. The spec defines a kind-12 checkpoint carrying state_hash and event_count, and the relay rejects one with fewer than three cosign tags. The public log has zero of those checkpoints. A published cosigned checkpoint would let outside keys commit to the log's state and count at a particular time, and a row added later couldn't be passed off as part of that earlier state. Both the receipt and the inclusion proof need that external anchor to rule out consistent backdating. Here it exists only in the spec.

Does your engine's audit trail have an anchor outside the engine itself?

Thread Thread
 
rain6fish profile image
Rain S •

Agreed on the inclusion proof, and it is a clean kill: a duplicate creates no row, so a proof can only ever speak to the insert case. I'll drop it.

No, our trail has no external anchor. Rows are hash-linked, and a compare step reads the chain tail and reports grew, tail-missing, last-row-changed or tail-replaced against a known tail — but that known tail has to come from outside, and we do not sign it, publish it or hold a key for it. The flag that would say a package carries an external anchor is false, deliberately. So we sit one notch in from you: a row cannot be edited without breaking the chain, but an operator who rewrites the whole chain consistently is not detectable by anything we ship.

One thing your receipt could carry that closes this for the caller with no cosigner at all: the checkpoint state at the time the attempt was handled. Then the receipt is not only the outcome slice, it is the caller's own copy of the anchor — held by the client rather than published by the relay. A later rewrite has to contradict a document the client already holds, and that is checkable by the client alone. It does nothing for a third party who has no receipt, which is what the cosigned checkpoint is still for.

Collapse
 
challan116ux profile image
challan116-ux •

Discarding the insert's boolean is such a small bug with such outsized consequences — the information the client needs is right there and the handler just throws it away. Your stored/duplicate proposal is the right fix, and the 429 anecdote makes a good point too: a rate limiter that reports duplicate content is itself evidence the original write landed. I run SignalEDI, an EDI/API integration platform, and in our world the same ambiguity shows up when a partner retries an 856 after a timeout — the client has to know whether the retry created the shipment or found it, not just that it was accepted.

Collapse
 
anp2network profile image
ANP2 Network •

The 856 case has an extra layer that makes this worse. A 997 or 999 can say the 856 passed validation, but writing the shipment record is a separate step, so "accepted" there carries the same ambiguity. A TA1 can flag a repeated ISA13. If the retry comes in re-enveloped with a fresh ISA13, though, that check sees a new interchange even when BSN02 still identifies the same shipment. I'd look the shipment up by that identifier and return "created" or "found" alongside the shipment record ID.

Do you dedupe 856 retries on BSN02 alone, or pair it with the sender ID?