Consumer Groups: Beginner

A Kafka consumer pulls. It sends a fetch request to the leader of each partition it reads, naming the offset it wants next, and the leader answers with the records from that offset forward. The call that does this is poll. One call can return hundreds of records from several partitions, or none at all when nothing new has arrived, and the consumer decides when to call again. The leader tracks nothing about what the consumer does with those records; it answers fetches. A broker acting as the group coordinator keeps track of which consumers read together and where each one has got to, and sections 1 and 2 cover it.

The log lesson showed that a record sits at an offset in a partition and that reading leaves it there. A consumer turns that into a loop: subscribe to a topic, call poll, handle every record in the batch, call poll again. The loop below is the whole job of a billing service that reads the orders topic and charges a card for each one.

Suppose the loop's 2nd poll returns 3 records. The consumer did not ask for offsets 4 and 5 of partition 0 by name; it asked for everything from its current position on each partition it holds, and the 2 leaders sent what they had. Within 1 partition the batch keeps offset order, 4 before 5. Across partitions there is no order, and partition 1's record may arrive before or after partition 0's.

After that batch the consumer remembers, in its own memory, that partition 0 is at 6 and partition 1 is at 13, and the next poll fetches from those positions. The position is in the consumer's memory alone; nothing has been saved to the broker. Saving it is the commit, which section 2 covers.

max.poll.records caps how many records a single poll hands to your code, 500 by default; the consumer fetches more than that in the background and parcels them out call by call. fetch.min.bytes and fetch.max.wait.ms tell the leader how long it may hold a fetch open while data piles up before it answers. With the defaults of 1 byte and 500 ms, the leader answers as soon as it has anything. max.partition.fetch.bytes limits how much of 1 partition comes back in 1 fetch, so a batch of 1 KB records from 1 partition tops out near 1,000 records in the buffer.

Between one poll and the next, the consumer does its work on the batch: it charges the cards, writes the rows, calls the downstream service. The broker does not watch this work; it sees only the next fetch. The consumer must call poll again within max.poll.interval.ms, 5 minutes by default. A batch of 500 records that take 1 second each finishes in about 8 minutes. That is past the deadline, so the group treats the consumer as gone. What the group does about it comes in a later lesson; for now, size the batch so the loop returns in time: 500 records at 1 second each needs max.poll.records under 300, or a handler 2 times faster.

A consumer joins a group by setting group.id. Every member of the group subscribes to the same topics, and the group as a whole reads every record once: each partition is handed to exactly 1 member, and that member reads it alone until the membership changes. A topic with 6 partitions read by a group of 3 gives each member 2 partitions. The same topic read by a group of 6 gives each member 1. A 7th member joins and finds nothing left to hand out. It keeps polling and getting empty batches until a partition frees up.

One broker acts as the group's coordinator. When a member starts, it tells the coordinator its group.id and the topics it subscribes to. The coordinator keeps the member list, and it calls a fresh deal whenever a member joins or leaves, cleanly or by crashing. Under the default protocol the deal itself is computed by 1 member, the group's leader, and handed to every member through the coordinator, so that every partition has 1 reader and no partition has 2. The default dealing spreads them as evenly as the counts allow: 6 partitions over 4 members come out 2, 2, 1, 1. How a member hands over a partition it was in the middle of reading is a later lesson; here the invariant is what matters: at any moment a partition has at most 1 reader in the group.

Groups do not share work with each other. A billing group and a fraud group both subscribed to orders each receive every record, and each keeps its own position. The split happens inside a group, never across groups. This is what lets 1 topic feed 10 independent systems with none of them seeing a subset, and it's why reading does not delete: 9 other groups may still need the record.

In the animation, every record with key cust-2 lands on 1 partition, and that partition belongs to 1 member, so 1 member sees all of that customer's orders in order. With 2 members each holds 2 partitions; with 5, 4 members hold 1 each and billing-e holds nothing. Use the crash control to stop billing-b: once the coordinator notices it is gone, its partitions are dealt to the others, and when it restarts it rejoins and the group deals again.

A consumer's position lives in its memory until it commits. A commit sends the group's coordinator 1 number per partition: the offset of the next record to read, which is the last processed offset plus 1. After processing offsets 7 through 9 of partition 0, the consumer commits 10. The coordinator appends that number to an internal topic named __consumer_offsets, keyed by the group and the topic-partition, so the committed position belongs to the group rather than to the member that sent it. Any member that holds partition 0 later asks the coordinator for the group's committed offset and starts at 10.

With enable.auto.commit=true, the default, the consumer commits on its own: during a poll, if auto.commit.interval.ms has elapsed since the last commit, 5 seconds by default, it commits the position its memory holds, and your loop never calls commit. The timer runs on the consumer's clock and takes no notice of whether your handler finished with the batch; what that means for a crash in the middle of one is the next lesson's first subject. With enable.auto.commit=false, nothing is committed until the loop calls commit, and the loop picks the moment: once a batch, once every 100 records, or right after a downstream write succeeds.

commit comes in 2 forms. The synchronous form blocks until the coordinator confirms, so the loop pauses for a round trip but knows the position was saved before the next poll. The asynchronous form sends the commit and returns at once, so the loop keeps polling and the confirmation, or the failure, arrives later. The Java client names them commitSync and commitAsync. A loop that commits once a batch usually affords the synchronous round trip; a loop that commits after every record usually cannot.

The committed offset answers where a consumer resumes. A group that has never committed has no answer: the coordinator looks in __consumer_offsets for the group's commit on that partition and finds nothing. The consumer then falls back to auto.offset.reset, a setting with 3 main values. latest, the default, starts at the end of the partition, so the consumer sees only records written after it first asked. earliest starts at the oldest offset the partition holds, 0 for a fresh topic, and reads everything. none raises an error and leaves the decision to you. Kafka 4.x also accepts a duration form, by_duration: followed by a span such as PT1H, which starts at the first record written within that span before now.

Take a topic orders with 5,000 records on 1 partition. A developer writes a new consumer with group.id=report-v1 and starts it, but no records arrive. The log shows the consumer joined its group and holds the partition, and poll keeps returning empty batches. The consumer is working as configured: with latest it began at offset 5,000, and no record has been written since. The 5,000 records are still there, as the log lesson promised, but this consumer stepped over all of them. The developer who expected a backfill wanted earliest; the developer running a live alerting service that must never replay a week of old events wanted latest. The default assumes the second case.

The setting applies only when the group has no valid committed offset on a partition. For a group that commits regularly, that is its first start, plus 2 rare cases: the committed offset points at records the broker has since deleted, or the group's commits expired while it had no members, which the tip below describes. On every other restart the committed offset wins. Changing the setting from latest to earliest on a group with a valid commit does nothing; the group resumes from its commit. To read from the start with such a group, either use a new group.id, which has no commits, or reset the group's offsets, a tool covered in a later lesson.

A group.id nobody has used is a new group, even when the code is the same. Rename report-v1 to report-v2 and redeploy, and the service starts under the reset setting again: with latest it skips everything written before the deploy, with earliest it reprocesses the topic from the start. Teams use that on purpose, giving a rebuilt service a fresh id and earliest to recompute a table from history, and by accident, when a config template changes the id and a live service silently skips an hour of events.

The producer writes 5 records before the consumer starts. Under latest, the consumer's first fetch asks for offset 5, and the 3 records the producer writes afterwards are the only ones it reads. Under earliest, the first fetch asks for 0, and it reads 8. Both runs commit after reading, so if either consumer restarts it continues from its commit and the setting no longer matters.

The command that prints the numbers ships with Kafka: bin/kafka-consumer-groups.sh --bootstrap-server broker-1:9092 --describe --group billing. It prints 1 row per partition the group reads, with the member holding each one.

CURRENT-OFFSET is the committed offset; the tool names it from the group's point of view. Partitions 0 and 1 sit within a few records of their ends, as does partition 3. Partition 2 is 7,766 behind while its reader, billing-b, keeps partition 3 nearly current, so that member is alive and committing. One reading can't say why partition 2 trails. Compare how fast each partition's log end grows between readings: if partition 2's grows far faster than the others, a hot key is sending it most of the traffic, the skew the producers lesson named; if it grows at their pace, suspect the handler, which is slow on partition 2's records or stuck on one of them. Either way, adding members won't help, because partition 2 can only ever have 1 reader.

A lag of 50,000 that was 500,000 an hour ago is a group catching up after an outage. A lag of 500 that was 50 yesterday and 5 the day before is a service losing ground slowly. When every CONSUMER-ID is blank, the group has no live members: its committed offsets stand still and lag grows at exactly the producers' rate until a member returns and resumes from them. Record the total and the per-partition lag on a schedule and graph it, because the describe command shows only the moment you run it.