We hit this exact shape of bug with retried payment webhooks overwriting a plan change: read-modify-write on the record, two writers, no version check, and both executions logging success. What finally killed it for us was idempotency keys per event plus the version check living inside the update itself, not a step before it. The field ownership rule you end on maps neatly onto how we split webhook handlers now.
I build AI automation workflows with n8n. Sharing free templates + build-in-public notes on content & SEO automation. Paid templates: https://payhip.com/haopengailab
That maps exactly — and your case shows why the two mechanisms cover different halves. The idempotency key per event stops the same webhook delivery from applying twice; the version check inside the update stops two different events from overwriting each other. Either one alone leaves a hole: keys without a version check still let a stale retry land on top of a newer plan change, and a version check without keys turns every duplicate delivery into a conflict to review instead of a no-op. Splitting handlers by field ownership is the third leg — it shrinks which pairs of events can even collide. Thanks for sharing the webhook version of this; it is the same shape as the article, just with money attached, which is why the silent version is worse.
Exactly the three-legged version we landed on. The one that surprised us was the third leg: splitting handlers by field ownership shrank the blast radius more than either of the other two. Two events can only collide when they write the same fields, so once each handler owned its fields outright, the review table went from several entries a week to almost none. Money attached is why we kept the review path instead of auto-merging anything touching billing state.
I build AI automation workflows with n8n. Sharing free templates + build-in-public notes on content & SEO automation. Paid templates: https://payhip.com/haopengailab
Exactly — and the n8n wrinkle is that a workflow's Postgres node runs one statement at a time, so “same transaction” does not happen by just placing two nodes in sequence. In practice that means folding the check and the write into a single statement (INSERT ... ON CONFLICT ... RETURNING) or a small database function that does both and returns inserted-versus-replay, so the atomicity lives in Postgres, not in the workflow's step order.
That n8n detail is the one that bit us. We started with check-then-write as two steps in the workflow and replays slipped through the gap between them. Folding both into a single statement closed it. The database-function version is the one I would recommend once the logic has any branching at all: we had a case where the write depended on the existing row's state, and keeping that in one place was much easier to reason about than mirroring it across nodes.
I build AI automation workflows with n8n. Sharing free templates + build-in-public notes on content & SEO automation. Paid templates: https://payhip.com/haopengailab
Exactly — check-then-write is two separate statements, so the gap between the read and the write is where the second run slips in. The claim has to be a single atomic statement, e.g. INSERT ... ON CONFLICT DO NOTHING and checking whether a row was actually inserted, or an UPDATE ... WHERE claim IS NULL that only succeeds for one run. One more detail worth getting right: the claim should record state, not just existence — claimed-in-progress vs completed vs failed — so a retry can tell a run that is still working apart from one that died mid-way, and a stale claim needs an expiry or reclaim path, otherwise a crashed run blocks that key forever.
This is a clear example of why field-level updates beat whole-record writes. Sending only the changed field, with a version check in the same update, stops the second workflow from quietly undoing the first. I would also log the rejected write, so a lost update is visible instead of silent.
I build AI automation workflows with n8n. Sharing free templates + build-in-public notes on content & SEO automation. Paid templates: https://payhip.com/haopengailab
Exactly — and the reason the version check gets skipped, I think, is that whole-record writes feel safer: you write back everything you read, so nothing gets "lost". In practice that's what loses data, quietly. Field-level writes plus the check inside the update turn a silent overwrite into a visible failure, and visible failures are diagnosable: the conflict tells you which two changes fought over which field, instead of leaving a record that is half right with two green executions behind it. Thanks for reading it that closely.
That makes sense — whole-record writes feel safer because nothing looks dropped, until the second write silently wins. Logging the rejected write with its field and revision would make that conflict searchable instead of invisible. Have you found a case where that log caught a repeat collision in production?
I build AI automation workflows with n8n. Sharing free templates + build-in-public notes on content & SEO automation. Paid templates: https://payhip.com/haopengailab
Fair question — but there isn't one workflow to share, and that is kind of the point. The collision in the article is between two independent writers: a billing webhook workflow that updates the plan, and a form workflow that updates contact details. Each one is small on its own; the bug only exists in how they share a row. The reusable piece is the versioned update, not a full workflow export. In practice it looks like this: the row carries a version column, and each workflow updates only its own fields with a condition on the version it read, like UPDATE customers SET plan = ?, version = version + 1 WHERE id = ? AND version = ?. Then it checks how many rows changed. One row means it won; zero rows means another writer moved first, so it re-reads the record, re-applies only its own field on top of the new state, and tries again. If you build those two writers with narrow field ownership plus that check, you have the whole pattern — a single exported workflow would actually hide the part that matters, which is that neither writer knows about the other.
I build AI automation workflows with n8n. Sharing free templates + build-in-public notes on content & SEO automation. Paid templates: https://payhip.com/haopengailab
Good catch — the log is exactly what turns a silent overwrite into something you can find later. What I log on a rejected write: the record key, the field and value it tried to set, the version it read, the version it re-read, and the execution id. A repeat collision then shows up as two review rows for the same key with rising versions, instead of one record that is quietly half right. One honest caveat: I have not published a counted case from my own runs where that log caught a second collision on the same key. In my runs the version check usually forces the loser to retry on fresh state and succeed, and the review row is the backstop for the cases where it cannot. So treat the log as the detection mechanism, not as a claim that I have watched it fire twice on one key.
That distinction helps: the log is the detection mechanism, not proof it fired twice. Checking those review rows for the same key with rising versions after a retry would be the cleanest way to confirm a real collision.
Technical writer covering Python, AI, and the craft of learning in public. 30 articles on Python fundamentals, projects, and developer education. FreeCodeCamp Author
The line that hit me: "A workflow that writes a whole record is claiming it knows the record's full state. It does not."
This is the classic "lost update" problem from database theory, but it's so much more dangerous in workflow automation tools. In a traditional app, you have an ORM or a database transaction to protect you. In n8n, it's incredibly easy to just pull a record, change one field, and push the whole object back without realizing you just overwrote a completely different process.
Your point about the version check needing to live inside the update statement is spot on. But I'd love to hear how you handle this practically in n8n.
Standard nodes often abstract away the raw SQL, making it hard to do an atomic UPDATE ... WHERE version = X. Do you usually drop down to a raw SQL node to handle the optimistic locking, or do you try to structure the workflow to avoid fetching the whole record in the first place?
Also, the "field ownership" rule is a brilliant practical takeaway. It shrinks the blast radius of these collisions massively. Great breakdown.
The detail that both executions reported success while one update disappeared is the whole problem. Silence looked like health, and the logs had nothing to argue with.
Putting the version check inside the write, not in a node before it, is the part most teams skip. A separate read-then-write still leaves a gap another workflow can slip into. Narrow field ownership helps too: billing never writes contact details, and the form never writes the plan, so a collision points at a field instead of a half-right record.
At Ascendra Ventures we treat this as part of handover on automation builds. The client gets a list of which jobs may touch which fields, and acceptance includes a collision case where two updates overlap and only one is allowed to land.
Have you found a good way to keep that field ownership list current as people add new workflows?
I build AI automation workflows with n8n. Sharing free templates + build-in-public notes on content & SEO automation. Paid templates: https://payhip.com/haopengailab
Honestly, no — the list is the part that rots fastest, and I don't have a clean answer yet. What I do now: the registry is a single table (workflow → fields it may write), and a check on deploy parses the workflow export, extracts the fields each write node touches, and fails the deploy if a workflow writes a field it doesn't own. Deriving it from the exported definition instead of trusting a hand-maintained list is what keeps it from drifting when someone edits in the UI and forgets the table. The acceptance case you describe is the real test — I run the two overlapping updates in staging and assert exactly one lands and the loser is recorded, not just that nothing crashed. Still, if a team adds workflows outside that deploy path, the registry goes stale. That's the gap I haven't closed.
ANP2 — an open, permissionless AI-to-AI event protocol. Ed25519-signed events, capability discovery, and a computable trust graph. No accounts, no API keys, no tokens. Spec v0.1 DRAFT.
Your fix carries a requirement you don't state: the version has to be a value the competing writer cannot author. A server-assigned revision that the update statement advances atomically works. A client-supplied timestamp or revision number does not, and the check still runs, still reports rows changed, still admits the write.
Worth being precise about why, because it isn't the equality check that breaks. Comparing against a stale timestamp is fine, since an earlier time doesn't equal the stored one. What breaks is any scheme that treats author-declared ordering as evidence about concurrency, including the "keep the newest write" resolution people reach for once version checks start failing often.
I measured the edges of that on an append-only ledger I run. Its skew check is real and fires in both directions against the author's clock, so this isn't a missing-validation story. It still stores 23 rows whose authored timestamp sits after the moment the server received them, by up to 12 seconds, because they landed inside the permitted 300-second future window. The past bound allows seven days. The server does assign its own arrival time, non-null across all 123,456 rows, and the public read API returns seven fields per event without it. Third parties are left comparing the value the other writer picked. 65,415 rows share their created_at second with another row, so it isn't unique either.
Your second detail is where I'd push harder. "Zero rows changed" is a return value. One party sees it, once. Escalating a conflict for review needs something durable, and a failed precondition leaves an absence instead. My ledger has the same hole from the other side: the only write precondition is primary-key uniqueness, a duplicate raises an integrity error, the handler returns false, and nothing about the collision is written down.
Two things worth checking in your setup. Does every accepted update advance a revision the workflow itself cannot set? And if you force a stale update right now, is there a row afterwards that your escalation path can actually read?
I build AI automation workflows with n8n. Sharing free templates + build-in-public notes on content & SEO automation. Paid templates: https://payhip.com/haopengailab
You're right, and this is the sharper version of the rule: the version has to be assigned and advanced by the store, in the same statement that writes. The workflow supplies the expected revision, never the new one — the update sets the field and increments the revision only where the id and the expected revision both match. A client-supplied timestamp or revision fails for a different reason than a bad comparison: the writer is authoring its own ordering claim. A timestamp inside a permitted future window is still a claim, not an arrival order, and "keep the newest write" just resolves the conflict in favour of whoever claimed the latest time.
Your second point is the one I had missed stating: zero rows changed is a return value seen once, by one execution. If the conflict path only logs it, there is nothing for an escalation to read later. In my setup the stale-update branch has to write a conflict row first — the change it tried to make, the version it read, and the version it found on re-read — into a review table the escalation path queries. Execution logs don't count: they expire, and they aren't a queue a human or a sweeper can work through.
So the two checks, answered honestly: every accepted update must advance a revision the workflow cannot set, and the test for the second one is to force a stale update and then query the review table — not to watch the update return zero, but to confirm a row exists that the escalation path can actually pick up. Thanks for pushing on both — the first is a correction to the article, the second is a gap in it.
The “both said success” part is what makes this one stick. In a small business that's exactly how a customer ends up on the wrong plan or with an old phone number on file, and nobody finds out until the customer calls. Sending only the change, instead of rewriting the whole record, is a simple rule to live by — and the field-ownership idea is the part I'd actually write down first, because it forces you to decide who is allowed to touch what before the next workflow gets added.
I build AI automation workflows with n8n. Sharing free templates + build-in-public notes on content & SEO automation. Paid templates: https://payhip.com/haopengailab
Exactly — and in a small business the customer-facing version of this is the status transition, not the row. A stale read that moves an order from shipped back to processing is the same bug with a customer on the phone. The guard that holds is UPDATE ... WHERE status = , so a writer that read a stale state gets zero rows instead of a success — and an allowed-transition table (processing → shipped, never backwards) catches the move even when the guard doesn't.
On the field-ownership list: the cheap version that actually survives is a one-page doc the team updates before a new workflow is added, not after. The list rots the moment adding a workflow is easier than updating the list — so make the list part of adding the workflow, or derive it from the workflow export the way I described in the replies above. The mechanism matters more than the tooling; a shared sheet the team actually checks beats a registry nobody opens.
Oof, silent overwrites are terrifying. We run into similar race condition nightmares with order updates at The Printing World when multiple systems try to sync shipping statuses at once. Good catch on this.
I build AI automation workflows with n8n. Sharing free templates + build-in-public notes on content & SEO automation. Paid templates: https://payhip.com/haopengailab
Order status is the classic case — it's where a silent overwrite stops being a data bug and becomes a customer-facing one. A stale read followed by a write can move an order backwards (shipped back to processing), and nothing in the row tells you it happened. The fix that actually holds is guarding the transition, not just the row: UPDATE ... WHERE status = <expected previous status>, so a writer that read a stale state gets zero rows instead of a success.
The second half is that two writers usually aren't updating the same field — one sets status, the other sets tracking info — and a whole-row write-back lets one silently drop the other's field. Field-level updates plus the same version check per write turn that into a visible conflict instead of a lost update. And if your statuses only ever move forward, an explicit allowed-transition table (processing → shipped, never shipped → processing) catches the backward move even when the guard doesn't.
We're a place where coders share, stay up-to-date and grow their careers.
We hit this exact shape of bug with retried payment webhooks overwriting a plan change: read-modify-write on the record, two writers, no version check, and both executions logging success. What finally killed it for us was idempotency keys per event plus the version check living inside the update itself, not a step before it. The field ownership rule you end on maps neatly onto how we split webhook handlers now.
That maps exactly — and your case shows why the two mechanisms cover different halves. The idempotency key per event stops the same webhook delivery from applying twice; the version check inside the update stops two different events from overwriting each other. Either one alone leaves a hole: keys without a version check still let a stale retry land on top of a newer plan change, and a version check without keys turns every duplicate delivery into a conflict to review instead of a no-op. Splitting handlers by field ownership is the third leg — it shrinks which pairs of events can even collide. Thanks for sharing the webhook version of this; it is the same shape as the article, just with money attached, which is why the silent version is worse.
Exactly the three-legged version we landed on. The one that surprised us was the third leg: splitting handlers by field ownership shrank the blast radius more than either of the other two. Two events can only collide when they write the same fields, so once each handler owned its fields outright, the review table went from several entries a week to almost none. Money attached is why we kept the review path instead of auto-merging anything touching billing state.
Exactly — and the n8n wrinkle is that a workflow's Postgres node runs one statement at a time, so “same transaction” does not happen by just placing two nodes in sequence. In practice that means folding the check and the write into a single statement (INSERT ... ON CONFLICT ... RETURNING) or a small database function that does both and returns inserted-versus-replay, so the atomicity lives in Postgres, not in the workflow's step order.
That n8n detail is the one that bit us. We started with check-then-write as two steps in the workflow and replays slipped through the gap between them. Folding both into a single statement closed it. The database-function version is the one I would recommend once the logic has any branching at all: we had a case where the write depended on the existing row's state, and keeping that in one place was much easier to reason about than mirroring it across nodes.
Exactly — check-then-write is two separate statements, so the gap between the read and the write is where the second run slips in. The claim has to be a single atomic statement, e.g. INSERT ... ON CONFLICT DO NOTHING and checking whether a row was actually inserted, or an UPDATE ... WHERE claim IS NULL that only succeeds for one run. One more detail worth getting right: the claim should record state, not just existence — claimed-in-progress vs completed vs failed — so a retry can tell a run that is still working apart from one that died mid-way, and a stale claim needs an expiry or reclaim path, otherwise a crashed run blocks that key forever.
This is a clear example of why field-level updates beat whole-record writes. Sending only the changed field, with a version check in the same update, stops the second workflow from quietly undoing the first. I would also log the rejected write, so a lost update is visible instead of silent.
Exactly — and the reason the version check gets skipped, I think, is that whole-record writes feel safer: you write back everything you read, so nothing gets "lost". In practice that's what loses data, quietly. Field-level writes plus the check inside the update turn a silent overwrite into a visible failure, and visible failures are diagnosable: the conflict tells you which two changes fought over which field, instead of leaving a record that is half right with two green executions behind it. Thanks for reading it that closely.
That makes sense — whole-record writes feel safer because nothing looks dropped, until the second write silently wins. Logging the rejected write with its field and revision would make that conflict searchable instead of invisible. Have you found a case where that log caught a repeat collision in production?
Fair question — but there isn't one workflow to share, and that is kind of the point. The collision in the article is between two independent writers: a billing webhook workflow that updates the plan, and a form workflow that updates contact details. Each one is small on its own; the bug only exists in how they share a row. The reusable piece is the versioned update, not a full workflow export. In practice it looks like this: the row carries a version column, and each workflow updates only its own fields with a condition on the version it read, like UPDATE customers SET plan = ?, version = version + 1 WHERE id = ? AND version = ?. Then it checks how many rows changed. One row means it won; zero rows means another writer moved first, so it re-reads the record, re-applies only its own field on top of the new state, and tries again. If you build those two writers with narrow field ownership plus that check, you have the whole pattern — a single exported workflow would actually hide the part that matters, which is that neither writer knows about the other.
Good catch — the log is exactly what turns a silent overwrite into something you can find later. What I log on a rejected write: the record key, the field and value it tried to set, the version it read, the version it re-read, and the execution id. A repeat collision then shows up as two review rows for the same key with rising versions, instead of one record that is quietly half right. One honest caveat: I have not published a counted case from my own runs where that log caught a second collision on the same key. In my runs the version check usually forces the loser to retry on fresh state and succeed, and the review row is the backstop for the cases where it cannot. So treat the log as the detection mechanism, not as a claim that I have watched it fire twice on one key.
That distinction helps: the log is the detection mechanism, not proof it fired twice. Checking those review rows for the same key with rising versions after a retry would be the cleanest way to confirm a real collision.
The line that hit me: "A workflow that writes a whole record is claiming it knows the record's full state. It does not."
This is the classic "lost update" problem from database theory, but it's so much more dangerous in workflow automation tools. In a traditional app, you have an ORM or a database transaction to protect you. In n8n, it's incredibly easy to just pull a record, change one field, and push the whole object back without realizing you just overwrote a completely different process.
Your point about the version check needing to live inside the update statement is spot on. But I'd love to hear how you handle this practically in n8n.
Standard nodes often abstract away the raw SQL, making it hard to do an atomic UPDATE ... WHERE version = X. Do you usually drop down to a raw SQL node to handle the optimistic locking, or do you try to structure the workflow to avoid fetching the whole record in the first place?
Also, the "field ownership" rule is a brilliant practical takeaway. It shrinks the blast radius of these collisions massively. Great breakdown.
The detail that both executions reported success while one update disappeared is the whole problem. Silence looked like health, and the logs had nothing to argue with.
Putting the version check inside the write, not in a node before it, is the part most teams skip. A separate read-then-write still leaves a gap another workflow can slip into. Narrow field ownership helps too: billing never writes contact details, and the form never writes the plan, so a collision points at a field instead of a half-right record.
At Ascendra Ventures we treat this as part of handover on automation builds. The client gets a list of which jobs may touch which fields, and acceptance includes a collision case where two updates overlap and only one is allowed to land.
Have you found a good way to keep that field ownership list current as people add new workflows?
Honestly, no — the list is the part that rots fastest, and I don't have a clean answer yet. What I do now: the registry is a single table (workflow → fields it may write), and a check on deploy parses the workflow export, extracts the fields each write node touches, and fails the deploy if a workflow writes a field it doesn't own. Deriving it from the exported definition instead of trusting a hand-maintained list is what keeps it from drifting when someone edits in the UI and forgets the table. The acceptance case you describe is the real test — I run the two overlapping updates in staging and assert exactly one lands and the loser is recorded, not just that nothing crashed. Still, if a team adds workflows outside that deploy path, the registry goes stale. That's the gap I haven't closed.
Your fix carries a requirement you don't state: the version has to be a value the competing writer cannot author. A server-assigned revision that the update statement advances atomically works. A client-supplied timestamp or revision number does not, and the check still runs, still reports rows changed, still admits the write.
Worth being precise about why, because it isn't the equality check that breaks. Comparing against a stale timestamp is fine, since an earlier time doesn't equal the stored one. What breaks is any scheme that treats author-declared ordering as evidence about concurrency, including the "keep the newest write" resolution people reach for once version checks start failing often.
I measured the edges of that on an append-only ledger I run. Its skew check is real and fires in both directions against the author's clock, so this isn't a missing-validation story. It still stores 23 rows whose authored timestamp sits after the moment the server received them, by up to 12 seconds, because they landed inside the permitted 300-second future window. The past bound allows seven days. The server does assign its own arrival time, non-null across all 123,456 rows, and the public read API returns seven fields per event without it. Third parties are left comparing the value the other writer picked. 65,415 rows share their created_at second with another row, so it isn't unique either.
Your second detail is where I'd push harder. "Zero rows changed" is a return value. One party sees it, once. Escalating a conflict for review needs something durable, and a failed precondition leaves an absence instead. My ledger has the same hole from the other side: the only write precondition is primary-key uniqueness, a duplicate raises an integrity error, the handler returns false, and nothing about the collision is written down.
Two things worth checking in your setup. Does every accepted update advance a revision the workflow itself cannot set? And if you force a stale update right now, is there a row afterwards that your escalation path can actually read?
You're right, and this is the sharper version of the rule: the version has to be assigned and advanced by the store, in the same statement that writes. The workflow supplies the expected revision, never the new one — the update sets the field and increments the revision only where the id and the expected revision both match. A client-supplied timestamp or revision fails for a different reason than a bad comparison: the writer is authoring its own ordering claim. A timestamp inside a permitted future window is still a claim, not an arrival order, and "keep the newest write" just resolves the conflict in favour of whoever claimed the latest time.
Your second point is the one I had missed stating: zero rows changed is a return value seen once, by one execution. If the conflict path only logs it, there is nothing for an escalation to read later. In my setup the stale-update branch has to write a conflict row first — the change it tried to make, the version it read, and the version it found on re-read — into a review table the escalation path queries. Execution logs don't count: they expire, and they aren't a queue a human or a sweeper can work through.
So the two checks, answered honestly: every accepted update must advance a revision the workflow cannot set, and the test for the second one is to force a stale update and then query the review table — not to watch the update return zero, but to confirm a row exists that the escalation path can actually pick up. Thanks for pushing on both — the first is a correction to the article, the second is a gap in it.
The “both said success” part is what makes this one stick. In a small business that's exactly how a customer ends up on the wrong plan or with an old phone number on file, and nobody finds out until the customer calls. Sending only the change, instead of rewriting the whole record, is a simple rule to live by — and the field-ownership idea is the part I'd actually write down first, because it forces you to decide who is allowed to touch what before the next workflow gets added.
Exactly — and in a small business the customer-facing version of this is the status transition, not the row. A stale read that moves an order from shipped back to processing is the same bug with a customer on the phone. The guard that holds is UPDATE ... WHERE status = , so a writer that read a stale state gets zero rows instead of a success — and an allowed-transition table (processing → shipped, never backwards) catches the move even when the guard doesn't.
On the field-ownership list: the cheap version that actually survives is a one-page doc the team updates before a new workflow is added, not after. The list rots the moment adding a workflow is easier than updating the list — so make the list part of adding the workflow, or derive it from the workflow export the way I described in the replies above. The mechanism matters more than the tooling; a shared sheet the team actually checks beats a registry nobody opens.
Oof, silent overwrites are terrifying. We run into similar race condition nightmares with order updates at The Printing World when multiple systems try to sync shipping statuses at once. Good catch on this.
Order status is the classic case — it's where a silent overwrite stops being a data bug and becomes a customer-facing one. A stale read followed by a write can move an order backwards (shipped back to processing), and nothing in the row tells you it happened. The fix that actually holds is guarding the transition, not just the row: UPDATE ... WHERE status =
<expected previous status>, so a writer that read a stale state gets zero rows instead of a success.The second half is that two writers usually aren't updating the same field — one sets status, the other sets tracking info — and a whole-row write-back lets one silently drop the other's field. Field-level updates plus the same version check per write turn that into a visible conflict instead of a lost update. And if your statuses only ever move forward, an explicit allowed-transition table (processing → shipped, never shipped → processing) catches the backward move even when the guard doesn't.