Kinesis Producers

Kinesis Producers

A producer for Amazon Kinesis Data Streams is an application that feeds user data records into a Kinesis data stream (also called data ingestion). The Kinesis Producer Library (KPL) makes it easier to construct producer applications by allowing developers to achieve high write throughput to a Kinesis data stream.

There are different methods to stream data into Amazon kinesis streams:

• Kinesis SDK
• Kinesis Producer Library (KPL)
• Kinesis Agent

Other third-party libraries include:

Spark, Log4J, Appenders, Flume, Kafka Connect, NiFi

Kinesis Producer SDK - PutRecord(s)

• PutRecord (one record) and PutRecords (many records) APIs are utilized.
• PutRecords leverages batching and enhances performance, resulting in fewer HTTP calls.
• AWS Mobile SDKs: Android, iOS, etc...

• Managed Amazon Web Services sources for Kinesis Data Streams:

  • AWS IoT
  • CloudWatch Logs
  • Kinesis Data Analytics

Use cases:
low throughput, higher latency, simple API, AWS Lambda

Kinesis Producer Library (KPL)

• Easy to use and highly configurable C++/Java library
• Used for building high-performance, long-running producers
• Automated and configurable retry mechanism
• Synchronous or Asynchronous APIs (better performance for async)
• Submits metrics to CloudWatch for monitoring.
• Batching (both turned on by default) – increase throughput, decrease cost:
  • Collect Records and Write to multiple shards in the same PutRecords API call.
  • Aggregate – increased latency.

Kinesis Producer Library (KPL) Batching

By inserting some delay using RecordMaxBufferedTime, batching efficiency can be impacted (default 100ms)

NOTE: When not to use the Kinesis Producer Library

• The KPL can incur an additional processing delay of up to RecordMaxBufferedTime within the library (user-configurable)
• Larger values ​​of RecordMaxBufferedTime result in higher packing efficiencies and better performance
• Applications that cannot tolerate this additional delay may need to use the AWS SDK directly

Kinesis Agent

Monitor Log files and sends them to Kinesis Data Streams
Java-based agent, built on top of KPL
Install in Linux-based server environments

Features:

• Write from multiple directories and write to multiple streams
• Routing feature based on directory/log file
• Pre-process data before sending to streams (single line, CSV to JSON, log to JSON)
• The agent handles file rotation, checkpointing, and retry upon failures
• Emits metrics to CloudWatch for monitoring

AWS Kinesis API - Exceptions

• Provisioned Throughput Exceeded Exceptions
• Happens when sending more data (exceeding MB/s or TPS for any shard)

• Make sure you don't have a hot shard (such as your partition key is bad and too many data goes to that partition) Solution:

  • Retries with backoff
  • Increase shards (scaling)
  • Ensure your partition key is a good one