DEV Community

Should webhook handlers trust the payload, or fetch the latest state from the API? Let's argue about it

delay.run on November 14, 2026

Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — delay.run A webhook arrives: order.updated, subscriptio...
Collapse
 
magnuszgr profile image
Magnus ZGR •

A practical split is to use the provider’s object version in a conditional upsert for mutable state, ignoring events older than the stored version; event IDs prevent duplicates but not reordering. If there’s no reliable version, store each event idempotently and coalesce a fetch by object ID. Keep one-time facts in an append-only event record so a later fetch can’t erase them.

Collapse
 
delayrun profile image
delay.run •

The append-only record for one-time facts is the piece I didn't spell out in the post, and it matters: a fetch tells you the subscription is active now, not that a payment went through at 14:02. Do you keep those records forever, or prune them once the state has settled?

Collapse
 
aiapikey profile image
HAOHAOnovai •

One addition: "event IDs prevent duplicates but not reordering" is the sentence I would put at the top of the design doc, because most implementations quietly treat dedupe as ordering.

The conditional upsert on the provider's version is what makes it actually safe, and the append-only side record is what stops a later fetch from rewriting history - that is the failure that only surfaces in an audit months later, when the one-time fact has been overwritten by a state snapshot that looked more recent.

Collapse
 
challan116ux profile image
challan116-ux •

Team doorbell, with one amendment: fetch the latest state, but keep the raw payload too. Re-fetching wins on ordering — a stale event can't overwrite newer state — but the payload is your evidence of what the sender believed at send time, and when the two disagree that difference is the bug report. In document-exchange work we keep both: the fetched state drives processing, the original bytes drive the argument about who changed what, when.

Collapse
 
delayrun profile image
delay.run •

Keeping the raw bytes as evidence is a good amendment, especially in document exchange, where the who-sent-what argument tends to show up weeks later. Do you keep them in the same database as the processed state, or push them off to cheaper storage after a while?

Collapse
 
aiapikey profile image
HAOHAOnovai •

Agreed, and I would make the storage shape part of the rule rather than a convention: keep the raw bytes with the received timestamp and the event id, immutable, next to the version you actually applied. Then "who changed what, when" is a query instead of a memory.

The metric that comes free is the disagreement rate - payload version versus fetched version at the same event id. It should sit near zero. A nonzero rate is either clock skew on the sender side or a sender that serialises state after emitting the event, and both are worth knowing before they become a dispute.

Collapse
 
krusenas profile image
Karolis •

If you wait and fetch once for several events, then fall back to the payload when the API is down, the single write can be stale. It is easy to miss because the debounce worked and the fallback worked, yet together they write old state. Anyone hit that in practice? Disclosure, I built Webhook Relay. It can store and retry incoming deliveries, but that does not fix a stale payload after an API outage.

Collapse
 
delayrun profile image
delay.run •

That combo is sneaky because each half passes its own test. I'd only let the fallback write go through if the payload's version is newer than what's stored, and otherwise park the event and fetch again once the API is back. Did you run into it after a real outage, or spot it in review?

Collapse
 
aiapikey profile image
HAOHAOnovai •

Yes, and the reason it is easy to miss is that both mechanisms are individually correct: the debounce coalesces events, the fallback keeps the write alive. They are jointly unsound because they read from different sources of truth at different times.

The fix I would reach for is making the version gate the only path to a write: the fallback payload still carries the event's version, so the same compare-and-apply runs whether the value came from the API or the body. A stale payload then fails the gate like any other old event, instead of being trusted because it arrived through the resilient branch.

Retrying deliveries helps the lost case, not the stale one - you are right that they are different problems.

Collapse
 
aiapikey profile image
HAOHAOnovai •

The useful split is not trust versus verify, it is what the handler needs: the delta, or the state. If the event carries a version, you can apply deltas safely by keeping the last applied version per object - idempotent, ordered, no extra read. If it does not, you are guessing about ordering anyway, so the fetch buys correctness you cannot get another way.

The cost of the fetch is easy to underestimate: one extra read per event, so a busy endpoint doubles its read QPS on that table at exactly the moment the write burst is happening.

Cheapest middle ground: fetch only on the event types that touch state you cannot reconstruct from the payload.

Collapse
 
delayrun profile image
delay.run •

Delta versus state is a cleaner way to frame it than trust versus verify. The doubled read load is a fair point too, since a burst is exactly when the provider is most likely to start rate-limiting your fetches. Which event types would go on your fetch-only list first?

Collapse
 
aiapikey profile image
HAOHAOnovai •

Fetch-only first, for anything where the payload is a trigger rather than the fact:

  1. money state - settlement, payout and refund transitions. A duplicated or stale "paid" payload is the one that provisions twice, and re-reading is cheap next to a reversal.
  2. entitlement changes - subscription and quota. The payload is a snapshot of a decision the billing system already committed to; the snapshot can be older than the decision.
  3. anything a human can override downstream. If support can revoke between event creation and consumption, the payload is already wrong by the time you read it.

Payload-first for append-only facts: disputes, delivery receipts, audit entries. There, the event IS the fact - and re-fetching can lose it if the provider's retention window is shorter than your retry window.