This article has four sections, and can jump ahead to the ones you're interested in:
Why they're set during topic creation
As we've previously see in the Kafka CLI commands section, there are two important parameters to define when creating a topic. These two impacts the performance and durability of your overall system.
- Partition Count
- Replication Factor
Let's use this diagram as an example.
Currently, we have three brokers and a topic with 2 partitions and 2 in-sync replicas. Now if we were change either of the two parameters,
- if we change the partition count to 3, the keys ordering guarantees will break.
- if we change the replication factor to 3, there will be pressure on your cluster which can lead to instability and performance decrease.
Increasing the replication factor maximizes the available brokers, which means each brokers will have to put in more work.
Partition Count
Here are a few points to remember:
- Each partition can handle a throughput of a few MB/sec
- Be sure to measure for your setup
- more partitions, more throughput; better parallelism
- more partitions means more consumers can run at a scale
- recall that no consumer will have the same topic partition
- more partitions are ideal in a large cluster
But:
- more partitions means more election to be done by Zookeeper
- more files are opened on Kafka
Thus, some guidelines are:
- for small cluster(less than 6 brokers), partitions should be twice the number of brokers
- for big cluster(more than 12 brokers), safe to have 1:1 ratio for partition and broker
- adjust for the number of consumers in parallel at peak throughput
- adjust for producer throughput (increase if super high throughput)
TEST! Every Kafka cluster will have different performance.
Replication Factor
Higher replication factor means:
- better resilience
- more replication
But:
- higher latencies if ack=all
- more disk space on the system
Thus, some guidelines are:
- should be at least 2, usually 3, max of 4
- Never set ISR=1 in production
- safe start with ISR=3
- if replication performance persists, get better hardware
Cluster Guidelines
- A cluster should not hold more than 2000 to 4000 partitions
- Maximum of 20,000 partitions across all brokers
- This is because when brokers crash, the Zookeeper performs leader elections for each partition
- Thousand of partitions means lot of elections
- If more partitions is needed, increase the number of brokers
- if more than 20,000 partitions is needed, follow the Netflix model and adapt Kafka clusters
Start at a reasonable partition count, test the performance, and increase accordingly.
If you've enjoyed this write-ups or if they somehow brought some value, I'll be glad to connect with you on Twitter!. 😃
Top comments (0)