A practical split is to use the provider’s object version in a conditional upsert for mutable state, ignoring events older than the stored version; event IDs prevent duplicates but not reordering. If there’s no reliable version, store each event idempotently and coalesce a fetch by object ID. Keep one-time facts in an append-only event record so a later fetch can’t erase them.
Hosted cron job service. Schedule HTTP requests on a cron, after a delay or right away. Every run logged, failed jobs retried, alerts by email or Slack.
The append-only record for one-time facts is the piece I didn't spell out in the post, and it matters: a fetch tells you the subscription is active now, not that a payment went through at 14:02. Do you keep those records forever, or prune them once the state has settled?
Building NovAI: one key for 114 models incl. Zhipu Flash free tier. I write about API economics - what a finished task costs, not what a token costs. Price by billing unit: aiapi-pro.com/chinese-ai-ap
One addition: "event IDs prevent duplicates but not reordering" is the sentence I would put at the top of the design doc, because most implementations quietly treat dedupe as ordering.
The conditional upsert on the provider's version is what makes it actually safe, and the append-only side record is what stops a later fetch from rewriting history - that is the failure that only surfaces in an audit months later, when the one-time fact has been overwritten by a state snapshot that looked more recent.
Team doorbell, with one amendment: fetch the latest state, but keep the raw payload too. Re-fetching wins on ordering — a stale event can't overwrite newer state — but the payload is your evidence of what the sender believed at send time, and when the two disagree that difference is the bug report. In document-exchange work we keep both: the fetched state drives processing, the original bytes drive the argument about who changed what, when.
Hosted cron job service. Schedule HTTP requests on a cron, after a delay or right away. Every run logged, failed jobs retried, alerts by email or Slack.
Keeping the raw bytes as evidence is a good amendment, especially in document exchange, where the who-sent-what argument tends to show up weeks later. Do you keep them in the same database as the processed state, or push them off to cheaper storage after a while?
Building NovAI: one key for 114 models incl. Zhipu Flash free tier. I write about API economics - what a finished task costs, not what a token costs. Price by billing unit: aiapi-pro.com/chinese-ai-ap
Agreed, and I would make the storage shape part of the rule rather than a convention: keep the raw bytes with the received timestamp and the event id, immutable, next to the version you actually applied. Then "who changed what, when" is a query instead of a memory.
The metric that comes free is the disagreement rate - payload version versus fetched version at the same event id. It should sit near zero. A nonzero rate is either clock skew on the sender side or a sender that serialises state after emitting the event, and both are worth knowing before they become a dispute.
If you wait and fetch once for several events, then fall back to the payload when the API is down, the single write can be stale. It is easy to miss because the debounce worked and the fallback worked, yet together they write old state. Anyone hit that in practice? Disclosure, I built Webhook Relay. It can store and retry incoming deliveries, but that does not fix a stale payload after an API outage.
Hosted cron job service. Schedule HTTP requests on a cron, after a delay or right away. Every run logged, failed jobs retried, alerts by email or Slack.
That combo is sneaky because each half passes its own test. I'd only let the fallback write go through if the payload's version is newer than what's stored, and otherwise park the event and fetch again once the API is back. Did you run into it after a real outage, or spot it in review?
Building NovAI: one key for 114 models incl. Zhipu Flash free tier. I write about API economics - what a finished task costs, not what a token costs. Price by billing unit: aiapi-pro.com/chinese-ai-ap
Yes, and the reason it is easy to miss is that both mechanisms are individually correct: the debounce coalesces events, the fallback keeps the write alive. They are jointly unsound because they read from different sources of truth at different times.
The fix I would reach for is making the version gate the only path to a write: the fallback payload still carries the event's version, so the same compare-and-apply runs whether the value came from the API or the body. A stale payload then fails the gate like any other old event, instead of being trusted because it arrived through the resilient branch.
Retrying deliveries helps the lost case, not the stale one - you are right that they are different problems.
Building NovAI: one key for 114 models incl. Zhipu Flash free tier. I write about API economics - what a finished task costs, not what a token costs. Price by billing unit: aiapi-pro.com/chinese-ai-ap
The useful split is not trust versus verify, it is what the handler needs: the delta, or the state. If the event carries a version, you can apply deltas safely by keeping the last applied version per object - idempotent, ordered, no extra read. If it does not, you are guessing about ordering anyway, so the fetch buys correctness you cannot get another way.
The cost of the fetch is easy to underestimate: one extra read per event, so a busy endpoint doubles its read QPS on that table at exactly the moment the write burst is happening.
Cheapest middle ground: fetch only on the event types that touch state you cannot reconstruct from the payload.
Hosted cron job service. Schedule HTTP requests on a cron, after a delay or right away. Every run logged, failed jobs retried, alerts by email or Slack.
Delta versus state is a cleaner way to frame it than trust versus verify. The doubled read load is a fair point too, since a burst is exactly when the provider is most likely to start rate-limiting your fetches. Which event types would go on your fetch-only list first?
Building NovAI: one key for 114 models incl. Zhipu Flash free tier. I write about API economics - what a finished task costs, not what a token costs. Price by billing unit: aiapi-pro.com/chinese-ai-ap
Fetch-only first, for anything where the payload is a trigger rather than the fact:
money state - settlement, payout and refund transitions. A duplicated or stale "paid" payload is the one that provisions twice, and re-reading is cheap next to a reversal.
entitlement changes - subscription and quota. The payload is a snapshot of a decision the billing system already committed to; the snapshot can be older than the decision.
anything a human can override downstream. If support can revoke between event creation and consumption, the payload is already wrong by the time you read it.
Payload-first for append-only facts: disputes, delivery receipts, audit entries. There, the event IS the fact - and re-fetching can lose it if the provider's retention window is shorter than your retry window.
We're a place where coders share, stay up-to-date and grow their careers.
A practical split is to use the provider’s object version in a conditional upsert for mutable state, ignoring events older than the stored version; event IDs prevent duplicates but not reordering. If there’s no reliable version, store each event idempotently and coalesce a fetch by object ID. Keep one-time facts in an append-only event record so a later fetch can’t erase them.
The append-only record for one-time facts is the piece I didn't spell out in the post, and it matters: a fetch tells you the subscription is active now, not that a payment went through at 14:02. Do you keep those records forever, or prune them once the state has settled?
One addition: "event IDs prevent duplicates but not reordering" is the sentence I would put at the top of the design doc, because most implementations quietly treat dedupe as ordering.
The conditional upsert on the provider's version is what makes it actually safe, and the append-only side record is what stops a later fetch from rewriting history - that is the failure that only surfaces in an audit months later, when the one-time fact has been overwritten by a state snapshot that looked more recent.
Team doorbell, with one amendment: fetch the latest state, but keep the raw payload too. Re-fetching wins on ordering — a stale event can't overwrite newer state — but the payload is your evidence of what the sender believed at send time, and when the two disagree that difference is the bug report. In document-exchange work we keep both: the fetched state drives processing, the original bytes drive the argument about who changed what, when.
Keeping the raw bytes as evidence is a good amendment, especially in document exchange, where the who-sent-what argument tends to show up weeks later. Do you keep them in the same database as the processed state, or push them off to cheaper storage after a while?
Agreed, and I would make the storage shape part of the rule rather than a convention: keep the raw bytes with the received timestamp and the event id, immutable, next to the version you actually applied. Then "who changed what, when" is a query instead of a memory.
The metric that comes free is the disagreement rate - payload version versus fetched version at the same event id. It should sit near zero. A nonzero rate is either clock skew on the sender side or a sender that serialises state after emitting the event, and both are worth knowing before they become a dispute.
If you wait and fetch once for several events, then fall back to the payload when the API is down, the single write can be stale. It is easy to miss because the debounce worked and the fallback worked, yet together they write old state. Anyone hit that in practice? Disclosure, I built Webhook Relay. It can store and retry incoming deliveries, but that does not fix a stale payload after an API outage.
That combo is sneaky because each half passes its own test. I'd only let the fallback write go through if the payload's version is newer than what's stored, and otherwise park the event and fetch again once the API is back. Did you run into it after a real outage, or spot it in review?
Yes, and the reason it is easy to miss is that both mechanisms are individually correct: the debounce coalesces events, the fallback keeps the write alive. They are jointly unsound because they read from different sources of truth at different times.
The fix I would reach for is making the version gate the only path to a write: the fallback payload still carries the event's version, so the same compare-and-apply runs whether the value came from the API or the body. A stale payload then fails the gate like any other old event, instead of being trusted because it arrived through the resilient branch.
Retrying deliveries helps the lost case, not the stale one - you are right that they are different problems.
The useful split is not trust versus verify, it is what the handler needs: the delta, or the state. If the event carries a version, you can apply deltas safely by keeping the last applied version per object - idempotent, ordered, no extra read. If it does not, you are guessing about ordering anyway, so the fetch buys correctness you cannot get another way.
The cost of the fetch is easy to underestimate: one extra read per event, so a busy endpoint doubles its read QPS on that table at exactly the moment the write burst is happening.
Cheapest middle ground: fetch only on the event types that touch state you cannot reconstruct from the payload.
Delta versus state is a cleaner way to frame it than trust versus verify. The doubled read load is a fair point too, since a burst is exactly when the provider is most likely to start rate-limiting your fetches. Which event types would go on your fetch-only list first?
Fetch-only first, for anything where the payload is a trigger rather than the fact:
Payload-first for append-only facts: disputes, delivery receipts, audit entries. There, the event IS the fact - and re-fetching can lose it if the provider's retention window is shorter than your retry window.