Groups Split Partitions
Concepts covered: Consumer groups
One partition, one reader at a time
A consumer joins a group by setting group.id. Every member of the group subscribes to the same topics, and the group as a whole reads every record once: each partition is handed to exactly 1 member, and that member reads it alone until the membership changes. A topic with 6 partitions read by a group of 3 gives each member 2 partitions. The same topic read by a group of 6 gives each member 1. A 7th member joins and finds nothing left to hand out. It keeps polling and getting empty batches until a partition frees up.
| members in the group | partitions each holds | idle members |
|---|---|---|
| 1 | 6 | 0 |
| 2 | 3 and 3 | 0 |
| 3 | 2 each | 0 |
| 4 | 2, 2, 1, 1 | 0 |
| 6 | 1 each | 0 |
| 7 | 1 each, 1 holds nothing | 1 |
The partition count is the ceiling on parallel readers. The 7th member adds no throughput; it adds a standby that takes over a partition when another member leaves. A group that needs more parallel readers needs a topic with more partitions, a number chosen when the topic is created. The split is also what keeps a key's records in order for a reader: all of customer 17's orders sit on 1 partition, that partition has 1 reader, so that reader sees them in the order they were written.
Who deals the partitions
One broker acts as the group's coordinator. When a member starts, it tells the coordinator its group.id and the topics it subscribes to. The coordinator keeps the member list, and it calls a fresh deal whenever a member joins or leaves, cleanly or by crashing. Under the default protocol the deal itself is computed by 1 member, the group's leader, and handed to every member through the coordinator, so that every partition has 1 reader and no partition has 2. The default dealing spreads them as evenly as the counts allow: 6 partitions over 4 members come out 2, 2, 1, 1. How a member hands over a partition it was in the middle of reading is a later lesson; here the invariant is what matters: at any moment a partition has at most 1 reader in the group.
2 groups, 2 full copies of the stream
Groups do not share work with each other. A billing group and a fraud group both subscribed to orders each receive every record, and each keeps its own position. The split happens inside a group, never across groups. This is what lets 1 topic feed 10 independent systems with none of them seeing a subset, and it's why reading does not delete: 9 other groups may still need the record.
- 2 consumers, same group.id
- Partitions split between them
- Each record read by 1 of the 2
- Adding the 2nd halves the 1st one's work
- Both share 1 committed position per partition
- 2 consumers, different group.id
- Each reads every partition
- Each record read by both
- Adding the 2nd changes nothing for the 1st
- Separate committed positions
- checkout (source, producer)
- orders (queue, Kafka topic); parallelism: 6 partitions
- billing (consumer, consumer group); parallelism: 3 members
- fraud (consumer, consumer group); parallelism: 2 members
- warehouse (consumer, consumer group); parallelism: 1 member
- checkout -> orders
- orders -> billing
- orders -> fraud
- orders -> warehouse
- 1 broker
- orders: 4 partitions, replication factor 1
- 5 members: 4 members hold 1 partition each; billing-e holds nothing
The billing group on orders, 4 partitions. Each partition goes to 1 member; with 5 members, 1 member holds nothing. Crash billing-b with the lever and watch its partitions move.
In the animation, every record with key cust-2 lands on 1 partition, and that partition belongs to 1 member, so 1 member sees all of that customer's orders in order. With 2 members each holds 2 partitions; with 5, 4 members hold 1 each and billing-e holds nothing. Use the crash control to stop billing-b: once the coordinator notices it is gone, its partitions are dealt to the others, and when it restarts it rejoins and the group deals again.
About this lesson
This section is part of the Consumer Groups: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.