gotchatypescriptModerate
Kafka consumer groups: parallelism equals the number of partitions
Viewed 0 times
kafka consumer grouppartitionparallelismrebalanceconsumer lagthroughput scalingidle consumer
Problem
Adding more consumer instances to a consumer group does not increase throughput beyond a certain point. Some instances sit idle. The team does not understand why.
Solution
In Kafka, one partition is assigned to at most one consumer within a group. Throughput scales with partition count, not consumer count. If you have 6 partitions and 10 consumers, 4 consumers will be idle.
Rule: max useful consumers per group = number of partitions. Set partition count based on your target throughput when creating the topic.
Rule: max useful consumers per group = number of partitions. Set partition count based on your target throughput when creating the topic.
Why
Kafka guarantees ordering within a partition by assigning each partition exclusively to one consumer. This is the mechanism — it is not configurable.
Gotchas
- Repartitioning an existing topic is possible but disrupts message ordering for keyed messages — plan ahead
- Partition reassignment during rebalance causes a pause in consumption — minimize unnecessary consumer churn
- Each partition rebalance triggers onPartitionsRevoked/Assigned callbacks — use them to commit offsets cleanly
- Consumer lag per partition is the key metric — alert when any partition's lag grows unboundedly
Context
Scaling Kafka consumers to increase message processing throughput
Revisions (0)
No revisions yet.