A gaming webhook worker should spend its latency budget on a token-bucket admission check, then treat HTTP 429 as a scheduled event instead of an immediate loop. Keep one idempotency key per delivery, honor Retry-After, and cap exponential backoff with jitter. That prevents duplicate match, purchase, and inventory events while recovering after a partner throttle.
Short answer: use the bucket for normal traffic and a durable delayed-retry queue for rejected traffic; a retry must never bypass the queue merely because its payload is old.
Decision record: invariants and boundaries
The worker owns four invariants. A token is consumed only when an outbound attempt is about to start. Every event keeps one stable key, such as game-8472:purchase:991, across retries. The scheduler persists the next eligible timestamp before acknowledging the queue message. The delivery ledger records the response class, attempt count, and last error code.
Those invariants define the failure boundary. A crash before a ledger write can produce an at-least-once attempt, so the receiver must deduplicate. A crash after the write but before acknowledgement can replay the queue message; that identifier makes replay harmless. This is a distributed-systems property, not a clever sleep() call.
Ship it.
For a gaming launch, latency and cost pull in opposite directions. A short retry window makes more requests and may invite more 429s. A long window protects the partner but delays an entitlement update. I set a maximum age per event type: presence can expire quickly, while a purchase confirmation deserves a longer horizon. Your mileage may vary because partner quotas differ.
| Option | Latency | Cost | Boundary |
|---|---|---|---|
| Fixed delay | Predictable, often slow in bursts | Extra attempts during throttling | Ignores partner timing |
| Exponential backoff + jitter | Fast first retry, spreads later load | Fewer synchronized retries | Needs cap and dead-letter policy |
| Token bucket + delayed queue | Smooth steady state | Attempts bounded by quota | Needs durable state and a clock |
How should a Node.js worker handle 429 backoff and delayed retries?
Parse Retry-After as seconds or an HTTP date, reject negative or absurd values, and combine the accepted value with a local ceiling. Without the header, use min(cap, base * 2 ** attempt) plus random jitter. Do not retry every 4xx response: malformed payloads and authentication failures need a terminal state, while 408, 429, and selected 5xx responses are policy choices.
The sequence below is Python, but the storage calls map directly to a Node.js/Express consumer. Picture a tournament launch where 40,000 inventory events enter in one minute: the bucket drains at the partner's agreed rate, a 429 moves one event to the delayed queue, and a second replica sees the same key and suppresses its request. If the enqueue succeeds but the acknowledgement is lost, the message returns; the ledger still says the key is pending, so the worker can safely examine the response and schedule only one next attempt. If the process dies between those writes, the outbox scanner repairs the gap. That chain is why I keep admission, persistence, and delivery as separate observable steps instead of hiding them in a retry helper.
import random
from datetime import timedelta
RETRYABLE = {408, 429, 500, 502, 503, 504}
def schedule(event, response, now, ledger, retry_queue):
key = event["idempotency_key"]
attempt = ledger.attempts(key) + 1
if response.status_code not in RETRYABLE:
ledger.finish(key, status=response.status_code, terminal=True)
return "terminal"
header = parse_retry_after(response.headers.get("Retry-After"), now)
ceiling = min(900, 2 ** min(attempt, 9))
delay = header if header is not None else ceiling
delay = max(1, min(delay, 900)) + random.uniform(0, 0.5)
eligible_at = now + timedelta(seconds=delay)
ledger.record_attempt(key, response.status_code, eligible_at)
retry_queue.enqueue(event, available_at=eligible_at)
return "delayed"
def send_once(event, http, bucket, ledger):
key = event["idempotency_key"]
if ledger.is_done(key):
return "duplicate_suppressed"
if not bucket.take(1):
return "reschedule_without_request"
return http.post("/partner/webhooks", json=event["payload"],
headers={"Idempotency-Key": key}, timeout=3)
Ordering matters. The bucket check sits immediately before the network call, so queue wait does not consume stale capacity. The ledger check comes first, so a redelivered message costs neither a token nor a request. Make the ledger write and enqueue operation transactional, or use an outbox table, because a split write can strand a retry.
One small trap: clocks. Delayed queues and database timestamps must use UTC, and a worker should tolerate modest skew. A five-second skew can turn a nominal 429 delay into a burst. Instrument token wait time, duplicate suppressions, response classes, and eligible_at - now; those distributions show where the latency budget goes.
The rejected design, and when is it valid?
I reject an in-process timer map for this workflow. It loses timers during deploys, duplicates work across replicas, and leaves no durable audit trail. It is valid for a best-effort cache refresh where an occasional miss is acceptable and no external side effect occurs. Webhook delivery is the opposite.
A dead-letter queue needs an explicit reason and final response class. Redact player data, and replay with the original idempotency key. Retrying a purchase with a new key can charge twice.
The catch is operational weight: durable ledgers, visibility timeouts, and metrics take more work than a loop with setTimeout. This design is not suitable when a partner has no idempotency mechanism and an ambiguous outcome is intolerable; use reconciliation or a human-reviewed outbox then.
Top comments (0)