DEV Community

Cover image for Difference between Kinesis Data Stream and SQS
Olawale Adepoju for AWS Community Builders

Posted on • Updated on • Originally published at dev.classmethod.jp

Difference between Kinesis Data Stream and SQS

What is the difference between Kinesis vs SQS?

Amazon Kinesis differs from Amazon's Simple Queue Service (SQS) in that it allows for real-time processing of streaming large data.

Kinesis Data Stream

AWS Kinesis Streams offers large-scale data input and real-time streaming data processing. It allows you to order records as well as read and/or replay them in the same order.

Kinesis Data Streams:

  • Data can be consumed many times
  • Data is deleted after the retention period
  • Ordering of records is preserved (at the shard level) – even during replays
  • Build multiple applications reading from the same stream independently (Pub/Sub)
  • “Streaming MapReduce” querying capability (Spark, Flink...)
  • Checkpointing is needed to track the progress of consumption (ex: KCL with DynamoDB)
  • Provisioned mode or on-demand mode
  • Stores record for 24 hours by default and can retain streaming data for up to 365 days
  • Can send stream records directly to services such as Amazon S3, Amazon Redshift, Amazon ElasticSearch, Splunk, AWS Lambda
  • Kinesis Provision Shards at 1MB/s produce and 2MB/s consumer

AWS SQS

AWS Simple Queue Service (SQS) offers a robust, highly scalable serverless hosted queue for storing messages and delivering data between application components.

SQS:

  • Queue, decouple applications
  • One application per queue
  • Records are deleted after consumption (ack/fail)
  • Messages are processed independently for standard queue
  • Ordering for FIFO queues (decreased throughput)
  • Capability to “delay” messages
  • Dynamic scaling of load (no-ops)
  • Can configure message retention period from 1 minute to 14 days, default is 4 days
  • Other services can be integrated through AWS Lambda
  • SQS does batch at 3000msg/sec and 30,000msg/sec in high throughput mode

Scenarios to use SQS vs Kinesis

SQS can be used in

  • Order processing
  • Image Processing
  • Auto-scaling queues according to messages.
  • Buffer and Batch messages for future processing

Kinesis Data Streams can be used in:

  • Rapid collecting and processing of log and event data
  • Real-Time metrics and reports
  • Real Time data analytics
  • Gaming data feed
  • Complex Stream Processing
  • Data Feed from IoT

Top comments (0)