Kafka Interview Questions for Data Engineers (2026)

Each consumer gets 3 partitions (12 / 4). When a consumer dies, a rebalance is triggered. The remaining 3 consumers split 12 partitions: two get 4 and one gets 4 (round-robin or range assignment). During rebalance, consumption pauses briefly. A strong answer mentions the rebalance protocol (eager vs cooperative), partition assignment strategies (range, round-robin, sticky), and the impact of session.timeout.ms and heartbeat.interval.ms on detection speed.

acks=0: producer does not wait for any acknowledgment. Fastest, but messages can be lost if the broker crashes before writing to disk. acks=1: producer waits for the leader to write the message. If the leader crashes before replication, the message is lost. acks=all: producer waits for all in-sync replicas to write. Slowest, but no data loss as long as min.insync.replicas is set correctly. Use acks=all for financial data, user events, and anything where loss is unacceptable. Use acks=1 for metrics or logs where occasional loss is tolerable.

Kafka guarantees ordering within a single partition. Messages with the same key always go to the same partition (via hash partitioning), so per-key ordering is guaranteed. There is no ordering guarantee across partitions. If you need global ordering, you need a single partition, which limits throughput to one consumer. A strong answer mentions that enabling retries with max.in.flight.requests.per.connection > 1 can break ordering unless you enable idempotent producers.

Consumer lag is the difference between the latest offset produced and the latest offset consumed. It is measured per partition. Monitor it with Kafka's consumer group describe command, Burrow, or the __consumer_offsets topic. Be concerned when lag is growing over time (consumers cannot keep up) or when lag spikes correlate with processing errors. A strong answer distinguishes between transient lag (a consumer restart) and sustained lag (throughput problem), and mentions scaling consumers or increasing partition count as solutions.

Reset the consumer group offsets to a timestamp three days ago using kafka-consumer-groups.sh with the reset-offsets flag and the to-datetime option. Alternatively, use the seek() method in the consumer API to set offsets per partition. The topic must have a retention period of at least three days (retention.ms). A strong answer mentions that the consumer application must be stopped before resetting offsets (or use a new consumer group), and discusses how downstream systems handle reprocessed (duplicate) data.

Compacted topics retain only the latest message per key. Kafka periodically removes older messages with the same key, keeping only the most recent value. Regular topics retain messages based on time or size. Use compacted topics for changelog/CDC data where you only need the current state of each entity (for example, user profile updates, product catalog). A strong answer mentions that tombstone messages (null value) delete a key, and that compaction is not immediate: old duplicates may be visible until the log cleaner runs.

Use a CDC connector (Debezium) as a Kafka Connect source to capture database changes. Enable idempotent producers on the connector. Use a stream processor (Kafka Streams or Flink) with the exactly-once configuration for any transformations. For the sink, use a Kafka Connect sink connector with upsert semantics (insert-or-update by primary key). A strong answer notes that exactly-once between Kafka and an external system requires idempotent writes to the target: either upserts, deduplication tables, or transactional commits with offset tracking.

The broker stops accepting writes to partitions on the full disk. If the broker is a partition leader, producers get errors and may fail or retry. Prevention: set retention.ms and retention.bytes to control topic size. Monitor disk usage with alerts at 70% and 80% thresholds. Use log.retention.check.interval.ms to control how frequently Kafka checks for expired segments. A strong answer mentions that different topics can have different retention settings, and that compacted topics need enough disk for both active and compacted segments during the cleaning process.

Kafka Streams is a library that provides stateful stream processing (windowing, joins, aggregations) with exactly-once semantics, built on top of the consumer API. A standalone consumer is simpler and appropriate for stateless transformations (filtering, enrichment from a cache, routing). Choose Kafka Streams when you need state management, event-time processing, or stream-table joins. Choose a standalone consumer when you just need to read, transform, and write without maintaining state. A strong answer mentions that Kafka Streams applications are just JVM processes (no separate cluster needed), unlike Flink or Spark Streaming.

Start with target throughput. If the topic needs to handle 100 MB/s and a single partition can handle 10 MB/s on your hardware, you need at least 10 partitions. Then double for headroom. Constraints: total partitions per broker should stay under ~4,000 (controller overhead), and you cannot decrease partition count without recreating the topic. A strong answer notes that more partitions hurts end-to-end latency (more files, more file handles), and that consumer parallelism is capped at the partition count.

Each partition is an append-only log on disk. Writes are sequential (extremely fast on both HDDs and SSDs). Reads are also sequential, leveraging the OS page cache. There is no random access by message ID; consumers read by offset (which maps to a position in the log). Implications: Kafka throughput is bounded by network and disk-write speed, not CPU. A strong answer notes that this is why Kafka can saturate a 10 Gbps NIC on commodity hardware.

Producers in the checkout service publish to an 'orders' topic, partitioned by customer_id (per-customer ordering). The Kafka cluster has 3+ brokers with replication factor 3. A Kafka Connect cluster runs JDBC source connectors capturing inventory changes, and S3 sink connectors archiving orders. Schema Registry stores Avro schemas for orders. Consumer groups: the inventory service updates stock, the email service sends confirmations, and a Flink job aggregates revenue. A strong answer covers data flow, replication, separation of compute (Connect) from storage (brokers), and which subsystems are stateful.

If a follower stops fetching from the leader for replica.lag.time.max.ms (default 30 seconds), it falls out of the ISR. The leader continues serving writes from the remaining ISR. If the partitioned follower had been the only ISR member besides the leader, and acks=all is required with min.insync.replicas=2, the producer will see NotEnoughReplicasException and writes pause. When the network heals, the follower fetches missed messages and rejoins the ISR. A strong answer covers the unclean.leader.election.enable tradeoff: false guarantees no data loss but may cause unavailability; true allows a stale follower to become leader and lose data.

Kafka is a distributed log: messages persist for a configured retention, and multiple consumer groups can read the same topic independently. Each consumer tracks its own offset. RabbitMQ is a traditional broker: messages are removed once consumed. Kafka excels at high-throughput event streaming, replay, and fan-out (many consumers reading the same data). RabbitMQ excels at task queues, priority routing, and complex dead-letter handling. A strong answer notes that the right choice depends on whether you need replay (Kafka) or per-message acknowledgment with routing logic (RabbitMQ).

Schema Registry enforces compatibility before allowing a new schema version. Backward compatible: new schema can read old messages (added optional fields are fine, removing fields is not). Forward compatible: old schema can read new messages (removing fields is fine). Full compatibility: both. Changing a field type is generally NOT compatible: producers register a new schema and Schema Registry rejects it under backward or full compatibility. A strong answer mentions compatibility levels (NONE, BACKWARD, FORWARD, FULL, plus _TRANSITIVE variants) and the deployment dance: bump consumers first under backward, producers first under forward.