Kafka Tutorial for Beginners

An event records that something happened. It carries a key and a value plus a timestamp, and it can have optional headers; Kafka also calls it a record or a message. Producers publish events, consumers subscribe to them, and the 2 never wait for each other.

A broker is a Kafka server that stores partitions and serves reads and writes. A cluster is a set of brokers, and each partition is replicated to several of them so that losing a machine does not lose the log. The documentation notes that a replication factor of 3 is common in production.

Retention decides how long events stay. The topic setting retention.ms defaults to 604,800,000 milliseconds, which is 7 days, and retention.bytes defaults to -1, so by default only time limits a log. Consuming an event never deletes it. That is the difference from a classic queue, and it is why replaying history is just reading from an earlier offset.

You need Java 17 or newer and a download of the Apache Kafka binaries from the Apache Kafka quickstart, which uses kafka_2.13-4.4.0.tgz. Kafka runs in KRaft mode, so there is nothing else to install: you generate a cluster ID, format the storage directory with it and start the server. The standalone option of the format command sets up a single-node cluster, which is what a laptop needs.

Open 3 terminals. The first runs the broker and stays busy, the second creates a topic and writes events, the third reads them. The commands in the next listing are the quickstart's own. The Apache project also publishes Docker images, so docker run -p 9092:9092 apache/kafka:4.4.0 starts a broker without a download.

The describe command prints 1 line for the topic and 1 per partition. Read the partition line from left to right: the partition number, the broker that is its leader, the brokers holding a replica of it and the in-sync replicas, written Isr. On a single local broker all 3 point at broker 0, because there is nowhere else to replicate to.

A write to a partition is not considered committed until all in-sync replicas have received it, and only members of that set can be elected leader. On a laptop the set has 1 member. On a production cluster it is the number you protect with the replication settings, min.insync.replicas among them.

A producer sends events in batches. With the defaults, it waits up to linger.ms of 5 milliseconds for more events to fill a batch of up to batch.size 16,384 bytes, trading a few milliseconds of delay for far fewer requests. compression.type defaults to none.

The safe settings are the defaults. acks is all, so a write counts only once the in-sync replicas have it, and enable.idempotence is true, so a retried send cannot write the same event twice. Retries are effectively unlimited, bounded by delivery.timeout.ms, which defaults to 120,000 milliseconds, 2 minutes.

The key decides the partition. Say a key hashes to 17. On a topic with 3 partitions it lands in partition 17 mod 3 = 2. If you add a partition, the same key now maps to 17 mod 4 = 1, so its new events are in a different log from its old ones. The documentation warns that after adding partitions, messages with the same key may be routed to different partitions, and that Kafka does not support reducing a topic's partition count. Choose the count with growth in mind, and choose a key such as a customer or device ID that groups the events whose order matters.

A consumer's progress in a partition is a single number, the offset of the next event it will read. A group stores 1 number per partition, so tracking what has been consumed is cheap, and rewinding means moving that number back.

When a consumer starts with no stored offset, auto.offset.reset decides where it begins, and the default is latest: it sees only events that arrive after it joins. That surprises beginners who write events, then start a consumer and see nothing. The console consumer in the quickstart is told to read from the beginning for exactly this reason.

A pair of timeouts keep a group healthy. A consumer that stops sending heartbeats for session.timeout.ms (45 seconds by default) is removed, and a consumer that goes more than max.poll.interval.ms (5 minutes by default) between calls to poll() is removed too. Either way its partitions move to the other members.

The bin/kafka-consumer-groups.sh tool, run with its describe option for a group, shows CURRENT-OFFSET and LOG-END-OFFSET for each partition, along with LAG. If the log end is 1,200 and the group's current offset is 1,150, the lag is 1,200 - 1,150 = 50 events. A lag that grows over time means the group reads slower than producers write, and the fix is more consumers up to the partition count or faster processing.

The design documentation names 3 delivery semantics. At most once means messages may be lost but are never redelivered. At least once means messages are never lost but may be redelivered. Exactly once means each message is processed once and only once. Which one you get depends on when the consumer commits its offset relative to processing: committing before processing means a crash loses events, and committing after means a crash repeats them.

Durability on the write side comes from replication. Take a topic with a replication factor of 3 and min.insync.replicas of 2. With acks=all, a write succeeds only if at least 2 replicas, the leader included, have it, so the topic keeps accepting writes with 1 replica down and refuses them when fewer than 2 remain. The default min.insync.replicas is 1, so raise it deliberately on topics that matter.

At least once plus idempotent processing is the common practical choice in pipelines. Writing a consumer's output so that a repeated event changes nothing is the same discipline as an idempotent data pipeline.

The default cleanup.policy is delete, which discards old segments when their time or size limit is reached. Setting it to compact changes the promise: Kafka retains at least the last known value for each message key, and compaction never reorders messages, it only removes some.

A message with a key and a null payload is treated as a delete for that key. A compacted topic therefore works as a changelog of current state, such as the latest address for each customer, which a new consumer can read from the start to rebuild a table.

Besides the command line tools, Kafka has 5 core APIs. Producer and Consumer handle sending and reading events, Admin manages topics, and Kafka Streams and Kafka Connect cover processing and moving data. You will write against Producer and Consumer when an application sends or reads events itself.

Kafka Connect runs reusable connectors that import data from external systems into topics and export topics to external systems, so moving a table or a file does not need custom code. The quickstart demonstrates it by copying lines from a text file into a topic and back out to another file. Kafka Streams is a client library for Java and Scala that processes the events in topics with transformations, including joins and windowed aggregations.

In a pipeline, Connect or a change data capture tool fills topics from source systems. Streams or another processing engine transforms them, and sinks write results to storage. How change capture feeds a topic is covered in what CDC is, and the place of a log like Kafka beside scheduled loads in batch vs streaming pipelines.

Kafka orders events within a partition only. If 2 events must be seen in order, they need the same key, and a topic with many partitions gives no order across keys. Reading the log is also not a query: finding 1 event by a field means scanning, so lookups belong in a store that consumers write to.

Partition counts only grow, and growing them changes where keys land, as the hash arithmetic for a key shows. A group's parallelism stops at its partition count. And the safe durability settings are a choice per topic: with a replication factor of 1, the local quickstart setting, a broker failure loses the partition.

Interviewers use Kafka to see whether you can reason about what breaks: why a key is chosen, what happens when a consumer is slower than the producers, and how you would avoid double counting after a restart. The Kafka interview questions cover those topics, and the data pipeline interview questions place Kafka inside the wider design.

To practice designing the pipeline around a topic, work through the data pipeline practice problems. The system design round guide shows how interviewers expect an event pipeline to be walked through.