Reading Consumer Lag

Concepts covered: Consumer lag

Lag is a subtraction per partition

Every partition carries 2 numbers a group cares about: the log end offset, 1 past the newest record the leader holds, and the group's committed offset. Lag is the difference. A partition whose log end is 8,420 and whose committed offset is 8,400 has a lag of 20: the group has 20 records it has not yet committed. Lag belongs to a partition and a group, so a topic read by 3 groups has 3 lags per partition, and a group reading 6 partitions has 6 lags, usually summed into 1 number on a dashboard.

The subtraction uses the committed offset, never the position in the consumer's memory. A consumer that has polled and processed 300 records since its last commit shows those 300 as lag until it commits. With auto-commit every 5 seconds, a healthy group's lag on a busy partition bounces between 0 and a few seconds of traffic; with a loop that commits once a batch, it bounces by the batch size.

Describing a group

The command that prints the numbers ships with Kafka: bin/kafka-consumer-groups.sh --bootstrap-server broker-1:9092 --describe --group billing. It prints 1 row per partition the group reads, with the member holding each one.

CURRENT-OFFSET is the committed offset; the tool names it from the group's point of view. Partitions 0 and 1 sit within a few records of their ends, as does partition 3. Partition 2 is 7,766 behind while its reader, billing-b, keeps partition 3 nearly current, so that member is alive and committing. One reading can't say why partition 2 trails. Compare how fast each partition's log end grows between readings: if partition 2's grows far faster than the others, a hot key is sending it most of the traffic, the skew the producers lesson named; if it grows at their pace, suspect the handler, which is slow on partition 2's records or stuck on one of them. Either way, adding members won't help, because partition 2 can only ever have 1 reader.

3 readings of partition 2, 10 minutes apart, tell the direction. The log end moved 6,000 in each gap, so the producers write about 10 records a second. The committed offset moved 2,400, then 2,400 again: the reader processes 4 a second and falls 3,600 records further behind every 10 minutes. At that rate the partition never catches up; it needs a faster handler, or its traffic needs spreading over more partitions by the producers, because a 2nd member can't read it.

Lag's direction matters more than its size

Lag steady or shrinking

• The group keeps pace with the producers
• The number is about a batch size or a commit interval
• A spike after a deploy that drains in minutes
• Nothing to do but keep watching

Lag growing

• The group reads slower than the producers write
• Every partition growing: too few members, or a slow handler
• 1 partition growing: a hot key, or 1 member stuck
• Every CONSUMER-ID blank: no member holds anything

A lag of 50,000 that was 500,000 an hour ago is a group catching up after an outage. A lag of 500 that was 50 yesterday and 5 the day before is a service losing ground slowly. When every CONSUMER-ID is blank, the group has no live members: its committed offsets stand still and lag grows at exactly the producers' rate until a member returns and resumes from them. Record the total and the per-partition lag on a schedule and graph it, because the describe command shows only the moment you run it.

Do

• Alert on lag that grows for 10 minutes, rather than on lag above a fixed number
• Read lag per partition before adding members
• Check CONSUMER-ID when lag grows on every partition at once
• Commit often enough that lag reflects processing, within a batch

Avoid

• Add members past the partition count expecting lag to drop
• Treat a lag of 0 as proof the sink received the records
• Compare lags across groups as if 1 number meant 1 thing
• Judge a group from 1 reading of the command

About this lesson

This section is part of the Consumer Groups: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.