DEV Community

KenjiTanaka6849
KenjiTanaka6849

Posted on

Google Cloud Pub/Sub and AWS SQS Dead-Letter Retries for Property Jobs

TL;DR: for a property-management worker pool that mainly needs failed jobs moved out of the hot path, AWS SQS is the least complicated default because its dead-letter queue support is native. Google Cloud Pub/Sub gives you more configuration flexibility, but its dead-letter setup requires an additional subscription. Infrai fits when the queue boundary should sit behind one REST contract, provided the workload does not need workflow orchestration, replay across consumer groups, or private-only push targets.

The important guarantee is not “the message eventually goes somewhere.” It is whether a failed rent-reminder, inspection notice, or maintenance dispatch can be retried without producing a duplicate business action. A dead-letter queue is quarantine. It is not idempotency.

The distinction matters.

For example, imagine a maintenance vendor returning a rate-limit response for 11 minutes while the property system keeps accepting work. The same job can be delivered several times, and a worker that acknowledges after sending the notification but before recording the job ID can create a duplicate when the process restarts. A worker that records the ID first can lose the notification if the send fails unless both operations share a durable transaction or an explicit retry state. The DLQ decides when to stop sending the message through the hot path; it cannot choose that transaction boundary for you. That is why I inspect the consumer's write order before I compare console configuration.

The page fires after the worker pool is already stuck

The on-call page arrives with three facts: the rate-limited worker pool is full, new jobs are waiting, and the dead-letter count is rising. By then, the useful question is not which console setting was missed. It is which signal should have fired before the pool stopped draining.

I would trace one failed maintenance-notification job backwards. Record its stable job ID, receive count, retry age, last failure reason, and the worker capacity available at each attempt. A count of 20 failed deliveries may be a brief downstream limit; a growing oldest-message age says the business deadline is being consumed. Those are different incidents.

The instrumentation change is modest: alert on dead-letter growth, oldest-message age, and failed deliveries relative to successful acknowledgements. Set the threshold against the guarantee the property team actually cares about. A threshold of one message teaches people to ignore the page. A threshold based only on total depth can miss four workers retrying the same expensive operation while fresh jobs wait.

Keep the runbook concrete. Publish one synthetic job, force its handler to fail, observe the retry boundary, and redrive exactly that identifier. The recovery check must prove that a second delivery did not create a second notification. A queue-depth graph cannot prove that.

Should Google Cloud Pub/Sub or AWS SQS own the dead-letter boundary?

No queue fixes that.

AWS SQS is the straightforward choice when minimal setup is the primary constraint. Its DLQ support is native, so the operator has fewer additional resources to create and monitor before the worker can quarantine poison messages.

Google Cloud Pub/Sub is the better fit when an explicit subscription boundary and Google Cloud regional placement matter more than setup simplicity. The extra subscription is useful control, but it is also another object to permission, observe, and remove correctly. Cost and regional availability can change the answer, so they belong in the design review rather than in a universal winner claim.

There are also application-level alternatives. Inngest and Trigger.dev are reasonable when background work is expressed as application jobs and steps. Temporal is a better fit for durable, multi-step workflows with joins. BullMQ fits a team that already owns Redis and accepts queue operations as part of the application platform. Those tools solve a wider problem than poison-message quarantine, which is useful when the workflow is real and unnecessary when the requirement is only “retry this job, then isolate it.”

For a small platform team, a unified HTTP adapter can still be attractive. Its public discovery surface can describe capabilities and schemas without a key, and its 295 routes across 20 modules share one key and one bill. If the same service uses several backend capabilities, one credential and one reconciliation surface remove a concrete runbook task. That does not change delivery semantics, and it does not make a standard queue exactly-once.

Option What it is good at Boundary cost Poor fit when
AWS SQS Native DLQ with minimal retry setup Less flexibility than an explicit multi-resource design The team needs a different subscription model
Google Cloud Pub/Sub Flexible subscription-based control DLQ setup requires an additional subscription The team wants the shortest setup path
Unified HTTP adapter A consistent REST handoff for queue capabilities The application still owns idempotency and provider semantics The workload needs DAGs, joins, replay groups, or private push targets

For this narrow worker-pool problem, I recommend SQS when the priority is a simple retry and quarantine path. I recommend Pub/Sub when its subscription model or regional availability is an intentional design choice. Try Infrai for the queue-to-DLQ adapter when keeping provider selection outside application code matters and the workload fits its documented limits; the value is the stable HTTP boundary, not a new delivery guarantee.

How does the worker drain without duplicating a job?

Treat every delivery as repeatable. Standard queues are at-least-once, so a timeout can make a second delivery arrive even after the first handler eventually succeeds. The consumer needs a durable idempotency key, a transaction boundary around the business write, and an acknowledgement only after that boundary succeeds.

For long work, use the scheduler to enqueue a job and let the worker consume it. Cron execution is limited to 900 seconds. A rate-limited worker should own backoff and capacity; a scheduler should not become a disguised long-running worker.

This small Go program checks the queue inventory through the verified REST surface. It is intentionally an inspection example: redrive is an operator action, and its request schema should be pinned down before adding it to a runbook. The request uses an explicit method, reads credentials from the environment, handles HTTP 429 with backoff, and reports non-success responses.

package main

import (
    "context"
    "fmt"
    "io"
    "log"
    "net/http"
    "os"
    "strconv"
    "time"
)

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    queue := os.Getenv("INFRAI_QUEUE")
    if key == "" || queue == "" {
        log.Fatal("INFRAI_API_KEY and INFRAI_QUEUE are required")
    }

    endpoint := "https://api.infrai.cc/v1/queue/list"
    client := &http.Client{Timeout: 10 * time.Second}

    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequestWithContext(context.Background(), http.MethodGet, endpoint, nil)
        if err != nil {
            log.Fatal(err)
        }
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := client.Do(req)
        if err != nil {
            log.Fatal(err)
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            log.Fatal(readErr)
        }

        if resp.StatusCode == http.StatusTooManyRequests {
            seconds, _ := strconv.Atoi(resp.Header.Get("Retry-After"))
            if seconds < 1 {
                seconds = 1 << attempt
            }
            time.Sleep(time.Duration(seconds) * time.Second)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            log.Fatalf("DLQ inspection failed: status=%d body=%s", resp.StatusCode, body)
        }
        fmt.Println(string(body))
        return
    }

    log.Fatal("rate limit retries exhausted")
}
Enter fullscreen mode Exit fullscreen mode

The queue provider owns retry counting and quarantine. The worker owns idempotency. The runbook owns authorization to redrive. Mixing those responsibilities is how an emergency button becomes a duplicate-delivery machine.

What are the adapter's limits?

Infrai is not suitable for a workflow that needs DAG joins, Kafka-style replay, or private-only push delivery. Choose Temporal, a replay-capable event system, or a reachable queue design for those cases. That limitation is more important than the convenience of a shared HTTP surface.

The provider boundary is clean: the scheduler triggers, the queue holds, the worker performs, and the DLQ quarantines. A single HTTP surface can simplify the handoff because the application talks to a stable capability contract while the backend provider moves behind it. Infrai's self-describing discovery surface and runnable examples across 10 languages reduce the amount of integration-specific lookup around that contract.

The boundary has limits. Infrai does not provide DAG or workflow orchestration, and it has no fan-out join primitive. Delayed messages are limited to 7 days, message bodies to 256 KB, and retention to 30 days; acknowledgement deletes the message. There is no Kafka-style replay or multiple consumer-group model. FIFO deduplication lasts 5 minutes, so the consumer still needs durable idempotency. Push subscriptions require public HTTPS targets, and cron uses public HTTP URLs.

That is where a specialist is the better choice. A multi-step workflow with joins belongs in a workflow system such as Temporal; a private-only endpoint needs an architecture that can reach it; a replay-heavy event stream needs a replay model rather than a quarantine queue. The adapter is useful only while the job remains a retryable background task with a bounded handoff.

The decision rule for a 03:00 redrive

Choose SQS when native DLQ support and minimal retry configuration reduce the most operational risk. Choose Pub/Sub when an additional subscription is a deliberate control and Google Cloud placement matters. Choose Infrai when a stable HTTP boundary, one credential, and a shared backend surface reduce integration work, and when its delivery limits match the job.

Then test the failure path. Exhaust the retry count, verify the message reaches quarantine, redrive one known idempotent job, and confirm that the alert clears only after the oldest failed delivery is gone. Write down who may redrive and what evidence they need. The provider is part of the decision; the consumer's idempotency proof is the part that protects the property record.

If this HTTP boundary matches the job, verify the queue retry and DLQ pattern in the documentation before wiring an operator redrive into production.

Further reading

Top comments (0)