Introduction to Amazon Managed Streaming for Apache Kafka (Amazon MSK)
Amazon MSK makes it easy to ingest and process streaming data in real-time with fully managed Apache Kafka.
APACHE KAFKA AT A HIGH LEVEL
What can fully manage Apache Kafka on AWS do:
- Allow you to create, update, and delete clusters
- MSK creates & manages Kafka brokers nodes & Zookeeper nodes for you
- Deploy the MSK cluster in your VPC, multi-AZ (up to 3 for HA) Automatic recovery from common Apache Kafka failures Data is stored on EBS volumes
- You can build producers and consumers of data
- Can create custom configurations for your clusters
- Default message size of 1MB
- Possibilities of sending large messages (ex: 10MB) into Kafka after custom configuration
MSK – Configurations
- Choose the number of AZs (3 – recommended, or 2)
- Choose the VPC & Subnets
- The broker instance type (ex: kafka.m5.large)
- The number of brokers per AZ (can add brokers later)
- Size of your EBS volumes (1GB – 16TB)
MSK – Security
Encryption:
- Optional in-flight using TLS between the brokers
- Optional in-flight with TLS between the clients and brokers
- At rest for your EBS volumes using KMS
Network Security:
- Authorize specific security groups for your Apache Kafka clients
Authentication & Authorization:
- Define who can read/write to which topics
- Mutual TLS (AuthN) + Kafka ACLs (AuthZ)
- SASL/SCRAM (AuthN) + Kafka ACLs (AuthZ)
- IAMAccessControl(AuthN+AuthZ)
MSK – Monitoring
CloudWatch Metrics:
- Basic monitoring (cluster and broker metrics)
- Enhanced monitoring (++enhanced broker metrics)
- Topic-level monitoring (++enhanced topic-level metrics)
Prometheus (Open-Source Monitoring):
- Opens a port on the broker to export cluster, broker, and topic-level metrics
- Setup the JMX Exporter (metrics) or Node Exporter (CPU and disk metrics)
Broker Log Delivery:
- Delivery to CloudWatch Logs
- Delivery to Amazon S3
- Delivery to Kinesis Data Streams
MSK Serverless
- Run Apache Kafka on MSK without managing the capacity
- MSK automatically provisions resources and scales compute & storage
- You just define your topics and your partitions and you're good to go!
- Security: IAM Access Control for all clusters
Kinesis Data Streams vs Amazon MSK
Top comments (0)