Where a New Group Starts
Concepts covered: Consumer groups
No committed offset, so the setting decides
The committed offset answers where a consumer resumes. A group that has never committed has no answer: the coordinator looks in __consumer_offsets for the group's commit on that partition and finds nothing. The consumer then falls back to auto.offset.reset, a setting with 3 main values. latest, the default, starts at the end of the partition, so the consumer sees only records written after it first asked. earliest starts at the oldest offset the partition holds, 0 for a fresh topic, and reads everything. none raises an error and leaves the decision to you. Kafka 4.x also accepts a duration form, by_duration: followed by a span such as PT1H, which starts at the first record written within that span before now.
| value | where the first poll starts | what the consumer sees |
|---|---|---|
| latest (default) | the end of the partition | only records produced after it starts |
| earliest | the oldest offset on the partition | every record the partition holds, then new ones |
| none | nowhere; poll raises an error | nothing until you assign a position by hand |
Why the default surprises people
Take a topic orders with 5,000 records on 1 partition. A developer writes a new consumer with group.id=report-v1 and starts it, but no records arrive. The log shows the consumer joined its group and holds the partition, and poll keeps returning empty batches. The consumer is working as configured: with latest it began at offset 5,000, and no record has been written since. The 5,000 records are still there, as the log lesson promised, but this consumer stepped over all of them. The developer who expected a backfill wanted earliest; the developer running a live alerting service that must never replay a week of old events wanted latest. The default assumes the second case.
The setting applies only when the group has no valid committed offset on a partition. For a group that commits regularly, that is its first start, plus 2 rare cases: the committed offset points at records the broker has since deleted, or the group's commits expired while it had no members, which the tip below describes. On every other restart the committed offset wins. Changing the setting from latest to earliest on a group with a valid commit does nothing; the group resumes from its commit. To read from the start with such a group, either use a new group.id, which has no commits, or reset the group's offsets, a tool covered in a later lesson.
A group.id nobody has used is a new group, even when the code is the same. Rename report-v1 to report-v2 and redeploy, and the service starts under the reset setting again: with latest it skips everything written before the deploy, with earliest it reprocesses the topic from the start. Teams use that on purpose, giving a rebuilt service a fresh id and earliest to recompute a table from history, and by accident, when a config template changes the id and a live service silently skips an hour of events.
bootstrap.servers=broker-1:9092
group.id=report-v1
auto.offset.reset=earliest- 1 broker
- orders: 1 partition, replication factor 1
- auto.offset.reset=earliest: The first fetch asks for offset 0
The group report-v1 has never committed. Under latest it starts at the end of the partition and sees only the 3 records produced later; under earliest it starts at offset 0 and reads all 8.
The producer writes 5 records before the consumer starts. Under latest, the consumer's first fetch asks for offset 5, and the 3 records the producer writes afterwards are the only ones it reads. Under earliest, the first fetch asks for 0, and it reads 8. Both runs commit after reading, so if either consumer restarts it continues from its commit and the setting no longer matters.
About this lesson
This section is part of the Consumer Groups: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.