Storage-first pattern using Serverless

Sometimes we have sensitive requests which we need to prepeare our systems to avoid losing it. Usually losing these requests mean that the company/product loses money or introduce issues in the product/application

Sensitive requests

• Payment requests
• Delivery requests
• Webhooks with external information

Non-sensitive requests

• List of users
• List of user-roles
• Endpoints for internal systems

The main difference between sensitive and non-sensitive request is that sensitive requests in general, you can not re-send it or it's more difficult in comparison with non-sensitive request

The idea behind of storage-first pattern, is basically to store the request information first ( in a high availability storage ) and process it later .

Why serverless is usefull in these cases ?

Using Serverless technologies like s3, sqs, kinesis, dynamodb gives you high availability without needing to pay for the provisioning. Using serverless storage you only pay per use.

we're going to explain here different use-cases and pattern, and talk a bit about pricing

Use cases

Here, there are 3 kind of implementation of storage-first pattern :

SQS-based Queueing

Ideal for: Payment processing, order fulfillment, webhook handling.

How it works:

• Request is immediately placed in an SQS queue.
• A separate process (Lambda function, ECS task, etc.) continuously polls the queue, processes messages, and performs necessary actions.

Benefits

• Decouples request ingestion and processing.
• Ensures high availability and fault tolerance.
• Scales automatically to handle spikes in requests.
• Retry mechanism

Pricing example

Processing per 1 M of requests

• SQS : $0.4
• API-GATEWAY (http): $1.00

• Total : $1.4*

• This is an estimate of the pricing because we're not taking care of Data Transfer

DynamoDB

Ideal for: Storing and retrieving sensitive request data quickly (e.g., payment details, delivery addresses)

How it works

• Critical request data is stored in DynamoDB with a unique identifier.
• Subsequent processes can look up and update data in DynamoDB with low latency using DynamoDB streams.

Benefits

• Fast and reliable data access.
• Serverless, scales automatically.
• Supports fine-grained access control.

Pricing example

Processing per 1 M of requests

• DynamoDB (write) : $1.25
• API-GATEWAY (http): $1.00

• Total : $2.25*

• This is an estimate of the pricing because we're not taking care of Data Transfer and Storage

S3

Ideal for: Storing raw request data for auditing, compliance, and analysis.

How it works:

• Requests are stored as objects in S3.
• S3 triggers events (e.g., object creation) to notify other services for further processing or analysis.

Benefits

• Durable and scalable storage.
• Cost-effective for long-term storage.
• Integrates with other AWS services.

Pricing example

Processing per 1 M of requests

• S3 put(standard) : $5
• API-GATEWAY (http): $1.00

• Total : $6*

• This is an estimate of the pricing because we're not taking care of Data Transfer and Storage

If you want to go deeper on AWS storage first pattern I highly recommend the following link