DEV Community

Cover image for The Transaction Only Covers Your Database. The Provider Key Is What Saves You
AheadKit
AheadKit

Posted on

The Transaction Only Covers Your Database. The Provider Key Is What Saves You

Two readers asked me the same question from opposite sides last week, and I have been running it ever since. itsvicmoreno asked it on my piece about claiming a delivery once: the processed-keys table stops the second copy, but what if the receiver crashes between writing the key and charging the card? sunnatulla_dev asked the matching half on the other piece: he had already tried it in his own head, one crash, two attempts, zero charges: the key written, the retry refused as a duplicate, and nothing ever charged.

Both of them were right to worry. A key and a charge are two separate writes, and a crash can land between them. Which order breaks first, and what actually saves you? I crashed the receiver on purpose, in both orders, against four different shapes, with the same $29 payment for the same order, and counted the charges from the receiver's own tables. The results are clear, and one of them made me change my mind about a fix I used to recommend without thinking.

Problem 1: key first, and the payment quietly never happens

The first crash is the one sunnatulla_dev described. The receiver writes the key, then dies before the charge. The sender sees the connection drop and tries again. The retry finds the key already in the table and gets a polite 200 duplicate. Nothing charges. Ever.

My run: 2 attempts, 0 charges. Exactly his shape, down to the numbers. The customer is never charged, the order sits unpaid, and nobody gets an error. The system looks healthy from every angle except the one that matters: no money moved. A duplicate refusal is only correct if the first attempt actually did the work, and here it did not. The key is a promise that the charge happened, and the crash broke the promise.

This is the quiet failure, and I think it is the one people underestimate. A double charge shows up in a bank statement and somebody shouts. A lost charge shows up weeks later, in a report, as revenue that never arrived, and by then nobody remembers the crash.

Problem 2: charge first, and the customer pays twice

Now flip the order. The receiver charges first, then dies before writing the key. The retry finds no key, so it charges again. Same $29, same order, second charge.

My run: 2 attempts, 2 charges. $58 for a $29 order. This is the order that breaks first, and itsvicmoreno's question bet exactly that: the charge lands and the key never does. The retry has nothing to check, because the only record of the first charge lives at the provider, and this receiver never asks the provider anything.

So both orders are broken, in opposite directions. Key first loses the payment. Charge first doubles it. If you have to pick a failure, pick the lost one — at least nobody's card was charged twice. But picking a failure is not a design. The fix has to cover both orders, not one.

Problem 3: the transaction saves your database, not your money

The first fix people reach for is the obvious one: put the key and the charge in one transaction, so they commit together or roll back together. I ran it. Crash before the commit, the whole thing rolls back, the retry charges exactly once. 2 attempts, 1 charge. It works.

But it works only inside your own database. The transaction covers the key and your local record of the charge. The charge itself happens at the provider, outside your database, outside your transaction, outside everything you can roll back. If the crash lands between the provider's charge and your mark-done, the transaction has nothing to say about it. The provider took the money. Your database says processing. The retry has to decide, and the transaction cannot decide for it.

I used to describe the single transaction as the answer to this problem. It is not the answer. It is a good answer to a smaller problem: keeping your own records consistent. The money lives somewhere else, and the fix has to reach there.

Problem 4: the lease has to be measured, and 30 seconds was a guess

The second fix is a status column with a lease. The key goes in as processing, with a time limit. A retry while the lease is alive gets a 409 and waits. After the lease expires, the row can be reclaimed and the charge retried. A crashed holder stops holding the row, eventually.

sunnatulla_dev ran this shape in his own test with a 30 second lease: the copy in flight got a 409, the crashed one got a 409, then it was reclaimed at 60 seconds, one charge each. My job was to check that number, and to set the lease from something real instead of 30 seconds picked out of the air.

So I measured first. One hundred real charge calls through the receiver, and I took the p99, the time almost every healthy call finishes inside. Run 1: p99 1.89 seconds, so the lease was 2.39 seconds. Run 2: p99 0.96 seconds, lease 1.46 seconds. The p99 moved between runs because handler time on this small machine includes scheduling stalls; run 1 even caught a 5.48 second stall. That is the point of measuring: the lease has to beat the longest a healthy handler can actually run on the machine it runs on, stalls included, plus slack. A lease picked from a round number cannot know that.

Then I crashed the lease arm and retried every half second. Run 1: three 409s, reclaimed 3.21 seconds after the crash, 1 charge. Run 2: two 409s, reclaimed 2.12 seconds after the crash, 1 charge. Not 60 seconds. The reclaim lands one retry after the lease expires, so the honest number is the lease plus your retry gap. His 60 seconds came from his 30 second lease plus his retry rhythm, not from the mechanism. With a lease set from p99, the crashed row is reclaimable roughly one lease after the crash, about two to three seconds here.

One more detail that matters: while the charge runs, the handler renews the lease, every third of a lease. Without that, a slow-but-alive charge gets reclaimed mid-flight and charges twice, which is the exact failure the lease was supposed to prevent. A lease without renewal is a timer that punishes the slow and rewards the dead.

Problem 5: the provider key is the only shape that survives both orders

The third fix is the one itsvicmoreno pointed at. Insert the key as processing, pass that same key to the provider as its idempotency key, then mark it done. A retry that finds processing does not charge again. It asks the provider: did this key already charge? If yes, mark it done and stop.

I crashed this shape after the provider charge, before the mark-done. 2 attempts, 1 provider charge, 1 local charge. The retry asked, the provider said already charged, nothing charged twice. Then I ran it in the charge-first order too: provider charge first, local key still pending, crash, retry, and the retry asked the provider first, before touching the local key table. Same result: 1 provider charge. It is the only shape in the test that stays at exactly one charge in both orders, including the window the transaction cannot see.

That is my claim, and I expect people to argue with it: the single transaction is the wrong place to put your trust, because the money does not live in your database. The status column with a measured lease is good and honest, and I would keep it as the backstop. But the thing that actually saves you from a crash between the key and the charge is the provider's own idempotency key, the same key on both sides, with the retry asking instead of charging. Everything else is bookkeeping about a charge that already happened somewhere else.

Have you ever been charged twice for one order, or had a payment quietly never go through after a crash? Which side did it break on, and what did the retry do: ask first, or charge first? I would like to hear what it was, because the retry is where the money is decided, and most of us never look at it until it has already decided wrong.


P.S. Everything I write stays free — that part doesn't change. But if this one was worth something to you, you can buy me a coffee ☕ (one-time or monthly, both work). It covers the API credits and hosting behind these experiments — and honestly, it's also the nudge that gets the next one started on the evenings I'd otherwise skip it. — Salam

Buy me a coffee

AheadKit on Gumroad — Beginner-Friendly Books & Packs

Top comments (13)

Collapse
 
ilragioniere profile image
Riccardo Vitali •

Answering your question honestly: all the double charges I own are ones I planted, and they are in a settlement-file ingest, not in a card charge. Same method as yours, different money.

One window your shape 5 leaves open: asking by key has a shelf life. Once the provider prunes the key, the only trace of the first charge is the identifier you own, carried with the charge, and if that lookup is a search rather than a read, it lags the charge by seconds. In that lag the only honest answer is unknown, and a retry that knows just charged / not charged gets it wrong in one of the two directions.

The test I would add to yours is the false positive: a provider that answers "already charged" when nothing was charged. That one skips a real charge forever, and no statement ever shows it. It is the quietest failure in the whole series.

The lag does have a floor under it, and it is the file. The provider's own settlement report is a charge record nobody prunes, dated, and independent of your key, which is why push can be lossy and still correct. A retry inside the lag can answer "unknown, hold" and let the file settle it when it lands. That is the piece I have been working, an ingest with thirteen tests and eight mutants, and the hardest faults there are not double writes: a record that dies mid-fetch, a record deleted after the write, a hand edit that keeps the file's size and line count. If you want to argue with the shapes: dev.to/ilragioniere/a-settlement-r...

Collapse
 
aheadkit profile image
AheadKit •

The false positive is the one I hadn't thought of — a double charge at least shows up on a statement, a skipped charge shows up nowhere. Going to run that one, a provider that lies about already-charged and count what it costs. The settlement file as the floor is the part I want to steal — is that what your thirteen tests are built around?

Collapse
 
ilragioniere profile image
Riccardo Vitali •

No. That's the honest answer and it is the useful part.

The thirteen are built around render(), not around the file. They assert: a settlement stores four fields and no verdict and no delta, so the verdict is derived at read time from the (commit, settlement) pair. A settlement carrying a different reference_rev is NOT_COMPARABLE, not FAIL, because a stale line cannot be compared to anything. Expiry carries an owner, and the owner bounds the comparison only if that owner can invalidate the basis (issuer yes, rail no). The clock and the invalidator set come from the caller with no defaults, and a missing clock must raise rather than fall back, which is the test that kills the mutant that supplies its own clock and therefore always passes. Provenance degrades: witnessed, issued, self_attested, and the writer cannot witness itself.

What the suite never does is read a settlement file. No provider, no fetch, no lag, no already-charged anywhere in it. The file is the floor operationally, because it is the one answer a party cannot talk itself out of, but the suite has a hole exactly where your new test goes.

So when you steal it: do not test that already_charged leads to a skip. Test the mutant that trusts the provider word as sufficient, and require a test to fail on it. Stub the provider to answer already_charged for a call that has no settlement line; assert the code goes and asks anyway; assert that with the line absent the answer is NO_SETTLEMENT with an age, not already_charged, and that with the line present and matching the answer is charged. The mutant to catch is three lines: if provider.already_charged: skip.

The caveat I gave widechaos holds harder for yours. Every fault is planted by the file that catches it, so a green run proves your path, not the provider. The one test that is not planted cannot run in CI: real sandbox account, real charge, prune the key, ask again, compare both answers to the settlement line. Name it in a comment inside the suite and leave it out of the run.

And on counting what it costs, the two errors look symmetric and are not. A false already_charged costs one charge, invisible on every statement, found by a customer. A false not_charged costs a double charge, a refund and a thread, but someone sees it. That asymmetry is the reason the provider's word can never be the only evidence.

Thread Thread
 
aheadkit profile image
AheadKit •

That three-line mutant is the whole test — if provider.already_charged: skip, and a real charge quietly never happens. The asymmetry is what sells it to me: the silent one costs a charge nobody ever sees. I'm stealing the shape exactly as you wrote it, including naming the sandbox test in a comment and leaving it out of CI. When you ran your thirteen, was there a mutant that surprised you by surviving, or did they all fall where you expected?

Thread Thread
 
ilragioniere profile image
Riccardo Vitali •

None survived. 8/8 caught in the run I published, and the no-op escaped as it must. The surprise was not a survivor, it was invisibility: two of the eight are byte-invisible to render(). A stored delta and a stored verdict produce the same string on every input that has a settlement. Behaviour cannot see them. Only the schema assertion kills them. Written the way I first thought about it, as output comparison, I would have reported 6/8 and called it clean, and the suite would have been certifying the one thing it could not see: what the record stores.

The one that taught me the most is the clock mutant: render() supplying its own clock when none is passed. It is only caught because there is a test that calls render() with no clock and requires it to raise. A test that always supplies a clock certifies a function that invents one. That is the same failure as a stub that always answers, one level up.

Two guards worth stealing with the shape. If a mutation's search string is not found in the source, exit with "mutation did not apply, so it proves nothing" rather than counting it as caught or escaped, because an unapplied mutation is a no-op and a no-op that scores is a broken harness. And print which test caught each mutant: "caught by test_expiry_owner_who_cannot_invalidate" is a sentence you can argue with; "8/8" is not.

Thread Thread
 
aheadkit profile image
AheadKit •

None surviving is the good news, the invisibility is the scary part — a suite certifying what it can't see is worse than a survivor, at least a survivor shows up. I'm stealing both guards, the unapplied-mutation exit especially, a no-op that scores is a broken harness. Is the clock mutant the one that changed how you write tests now, or was it the byte-invisible pair?

Thread Thread
 
ilragioniere profile image
Riccardo Vitali •

The clock one, but not for the reason it looks.

The byte-invisible pair changed what I assert: state that is stored has to be asserted as stored, never via render. That is a one-line fix, a schema check on the keys instead of a hundred output comparisons.

The clock changed how I call. A test that supplies a clock in every call is not testing the clock parameter, it is testing the code path that a clock enables. So the rule I now apply mechanically: for every dependency a function can take, exactly one call where it is absent and the expected result is a raise. Not a default, not None, a raise.

Its cousin is the one I still catch myself on. A parameter supplied in 100 percent of the calls in a suite is an untested parameter, and the suite reports it as covered. Counting how often an argument appears in your tests is cheap and it finds the stubs you forgot you had written.

Which is why your unapplied-mutation exit earns its keep. Same failure one level up: a mutation that never applied and a parameter that is never absent both hand you a green run with nothing underneath it.

Thread Thread
 
aheadkit profile image
AheadKit •

The clock one, then — and that rule about a parameter in 100% of the calls being untested is the one I'm stealing. Same shape as the unapplied mutation, just one level up: green run, nothing underneath. Do you count that mechanically across a suite, or is it more of a smell you check when something feels too green?

Thread Thread
 
ilragioniere profile image
Riccardo Vitali •

In mine it was a smell, and the test that made it one was the clock. Eight mutants, all killed, and I would have called the run clean. Then I tried to write the clock mutant and found the suite could not see it by any assertion about behaviour. The only thing that killed it was a test that passes no clock at all and requires a raise. Same for a parameter a suite supplies in every call: the green run says nothing about the world where it is missing.

A mechanical count is cheap and I have not built it yet. Call sites per parameter, and when the count of omitting calls is zero the suite exits "proves nothing" for that parameter, the same rule I give an unapplied mutation. The trap in the mechanical version is that presence of an omission is not coverage of it. A parameter can be left out in five calls that all assert the default and still never meet the absent-and-required case. So the census has to pair each omission with the expected result, or it counts the word "nothing" and files it as a test.

If you want that run over your own suite instead of argued about: twenty-five dollars, one suite. I read the call sites, list every parameter supplied in all of them, write the mutant that catches each one, and report which of your existing tests kills it. A report, not a library. Say the word and I send the link.

Collapse
 
itsvicmoreno profile image
Vic Moreno •

Thanks for crashing it in both orders, that 2 attempts, 1 provider charge is the number I didn't have. One window the provider key still leaves open: Stripe's docs say an idempotency key can be pruned once it is at least 24 hours old, so a retry that waits past that (long outage, a row reclaimed the next day) charges again. Do you cap retry age below the provider's key lifetime, or look the charge up by your own order id first?

Collapse
 
aheadkit profile image
AheadKit •

Didn't know about the 24-hour prune — that's a real hole in the shape as I ran it, my retries all landed inside the window. My instinct is the order-id lookup, since a cap still leaves you guessing at the provider's clock. Going to run the expired-key retry and see which one actually holds. Have you ever seen a retry land past the prune in the wild, or is this all from the docs?

Collapse
 
itsvicmoreno profile image
Vic Moreno •

From the docs, not from the wild: Stripe's docs, for one, say keys can be pruned once they are at least 24 hours old, and a reused key after that is treated as a brand new request. So the place I'd expect it is a retry path that can sleep past a day, like a dead-letter replay after a weekend outage. Agree on the order-id lookup: search the provider by your own order id before retrying, and the provider's clock stops mattering.

Thread Thread
 
aheadkit profile image
AheadKit •

The dead-letter replay after a weekend outage is the exact shape I had in mind — that's the retry path that actually sleeps past a day. Order-id lookup first, and the provider's clock stops mattering. When you run that replay, do you cap how old a dead letter can be before refusing it outright, or does the lookup make the age irrelevant?