Kafka vs Kinesis

A Kafka topic is a set of partitions, each an ordered, replicated log on a leader broker and its followers. A producer that sends a key hashes it to pick the partition, so every record for that key lands in the same log in the order it was written. Consumers in a group split the partitions between them and track their position as 1 offset per partition, which they can rewind to replay.

A Kinesis stream is a set of shards. Each record carries a partition key, which Kinesis hashes with MD5 into a 128-bit number; every shard owns a contiguous range of that number line, and the record goes to the shard whose range holds its hash. Records in a shard carry sequence numbers, and consumers built on the Kinesis Client Library (KCL) keep their checkpoints and shard leases in DynamoDB tables.

On paper the models match. In practice the Kinesis shard is also the unit you pay for and the unit that throttles: a producer that pushes more than 1 MB/s or 1,000 records a second into 1 shard gets ProvisionedThroughputExceededException and must retry. A Kafka partition slows down only when its leader broker runs short of CPU, or of disk or network throughput, and the fix is a bigger or additional broker rather than a quota.

A provisioned stream needs enough shards to cover writes, counted in both bytes and records a second, and to cover reads. Each shard takes 1 MiB of writes a second and serves 2 MiB of reads a second, and every consumer that reads with GetRecords shares that read capacity with the others, so a second consumer doubles the read requirement.

Take 5,000 events a second at 2 KiB each, read by 2 consumers. Writes are 10,000 KiB a second, which needs 10 shards. Reads are 20,000 KiB a second, which also needs 10. The record rate needs only 5. The stream needs 10 shards, and since partition keys never spread perfectly, leave headroom of a quarter or more on top.

Consumers change the arithmetic. Enhanced fan-out gives each registered consumer its own 2 MB/s per shard, pushed over HTTP/2, so read capacity stops being shared; it costs extra per consumer-shard hour and per gigabyte. Kafka has no per-partition read quota at all: any number of consumer groups can read the same partition, limited only by broker network and disk.

Both order records within 1 partition or shard and nowhere else, so the partition key decides what is ordered. Key by order_id and every event of an order is in sequence; events of different orders interleave in whatever order they arrive.

Delivery is where they part. Since version 0.11, Kafka has an idempotent producer, which uses sequence numbers so a retry cannot write a duplicate, and transactions, which commit writes to several partitions and the consumer's offsets atomically. A job that transforms records from 1 topic into another can therefore be exactly once end to end inside Kafka, which is what Kafka Streams and Flink's Kafka sink rely on.

Kinesis is at least once, and AWS documents the 2 sources of duplicates. A producer that times out before its acknowledgement and retries writes the record twice, under 2 sequence numbers. A consumer restarted by a crash or a deploy resumes from its last checkpoint and processes records again, and so does one restarted by a reshard. Deduplication has to happen in the sink. Embed a unique id in each record and upsert on it, or write files whose names are fixed by the shard and the first sequence number, so a retry overwrites instead of duplicating.

In an interview, say where duplicates come from and where you remove them. If you claim exactly-once on Kinesis, name the idempotent sink that provides it; without that sink the answer is wrong, however the rest of the design looks.

A Kinesis stream keeps records for 24 hours by default; you can raise retention to 365 days and pay for every hour past the first day. Replay means asking for a shard iterator at a timestamp or sequence number, per shard. Long retention on Kinesis is priced per gigabyte-month, so most teams land the stream in S3 and replay history from there.

Kafka's broker default is 168 hours, and retention can be any duration or size, or unlimited, bounded by the disks you pay for; tiered storage moves older segments to object storage so a long retention no longer means large broker disks. Replay is resetting a consumer group's offsets, a single command across all partitions.

Kafka also has log compaction, which Kinesis has no equivalent for: a compacted topic keeps at least the latest record for every key, so it works as a changelog that a new consumer can read from the start to rebuild current state. CDC pipelines lean on it, keeping the latest row image per primary key without keeping every update forever.

Every Kinesis consumer that polls with GetRecords shares a shard's 2 MB/s and its 5 read calls a second. AWS measures an average propagation delay of about 200 ms with 1 such consumer, rising to about 1,000 ms with 5. Enhanced fan-out pushes records to each registered consumer with about 70 ms of delay, and a stream allows 20 registered consumers, or 50 under On-demand Advantage.

A Kafka topic serves any number of consumer groups, and adding one costs only broker network and disk reads, which mostly hit the page cache for recent data. That makes Kafka the more natural hub when many teams read the same events: the warehouse loader and a fraud model can read 1 topic at their own pace, alongside a search indexer and an audit archive.

Consumer scaling is capped the same way on both. 1 KCL worker per application reads each shard, and 1 consumer per group reads each partition, so no consumer can spread its work wider than the partition or shard count. Rebalances move that ownership around; Kafka 4.0 made the newer consumer group protocol, which rebalances incrementally on the broker, generally available.

Kinesis prices are public, so the arithmetic is exact. Take 100,000 records a second of 1 KiB, roughly 100 MB/s, read by 1 consumer, every hour of a 730 hour month at US East prices. Provisioned mode needs 100 shards for the record rate: $1,095 of shard hours plus 262.8 billion PUT payload units at $0.014 per million, $3,679, for about $4,774 a month.

On-demand bills the same 250,626 GB by volume. On-demand Standard charges $0.08 per GB in and $0.04 per GB out, about $30,104 a month. On-demand Advantage, an account-level commitment of at least 25 MiB/s, bills $0.032 per GB in and $0.016 per GB out, which comes to about $12,030. For flat traffic, provisioned shards cost a fraction of on-demand; on-demand is worth its price when traffic is spiky enough that provisioning for the peak would leave most shards idle.

2 details move these numbers. On-demand rounds every record up to 1 KiB, so 200 byte events are billed as 1 KiB each, and provisioned PUT payload units are 25 KB, so batching small events into larger records with the Kinesis Producer Library cuts the PUT bill sharply in both modes.

Kafka has no list price, because you pay for what you run: brokers sized for the peak, disks for retention times the replication factor, and on AWS the data transfer between Availability Zones that replication and cross-zone consumers generate. A managed service such as MSK or Confluent Cloud adds its margin in return for running it. Compare the 2 on the same steady volume with your own broker quote, and count the engineers who operate the cluster as part of its price.

With Kinesis your work is choosing partition keys that spread load and sizing or scaling the shards, plus handling throttling and duplicates in your producers and consumers. You run no brokers, so their disks and upgrades are not yours to own, and neither is replication.

Self-managed Kafka leaves you the whole cluster. You size the brokers and set the replication factor along with min.insync.replicas. Disk growth and rolling upgrades are yours, as is partition reassignment when you add brokers, and you monitor under-replicated partitions and consumer lag. Kafka 4.0, released in March 2025, removed ZooKeeper, so a cluster now keeps its metadata in its own KRaft controllers and there is 1 less system to run.

Between the 2 sit the managed Kafka services. Amazon MSK runs the brokers inside your VPC while you still choose partitions and broker sizes; Confluent Cloud and Redpanda Cloud sell throughput and take the cluster off your hands. Redpanda is a separate implementation of the Kafka API in C++. Kafka clients work against it unchanged, and it handles partitions and keys the way Kafka does, partition expansion included.

Use Kinesis when the platform is AWS, throughput is in the tens of MB/s or is hard to predict, a replay window of a day to a week covers your recovery plans, and the consumers are a handful of AWS services such as Lambda and Firehose, or Managed Service for Apache Flink. The team ships a stream on day 1 and never pages for a broker.

Use Kafka when many teams build on the same stream: exactly-once pipelines, compacted changelog topics for CDC, retention long enough to rebuild a table from the log, dozens of consumer groups, Kafka Connect for sources and sinks, or a deployment across clouds and data centers. If you want those semantics without staffing a cluster, get them from MSK on AWS, or from Confluent Cloud or Redpanda.

A useful test is the spike you expect. If traffic can double in minutes with no warning, Kinesis in on-demand mode takes up to 2 times its recent peak with no action from you. If the peak is known, Kafka sized for it costs less at volume and never throttles below that size. The batch vs streaming guide covers the earlier question of whether the pipeline should stream at all.

The comparison comes up in the design round, rarely as a trivia question and usually as a follow-up: you chose a stream for clickstream or CDC events, and the interviewer asks why that one, how you would key it, what happens when traffic quadruples, and how you avoid double counting. Strong answers tie each choice to a number: the shard or partition count from the throughput, the retention from the replay plan, the delivery guarantee from the sink.

Practise the design end to end with the data pipeline practice problems, which have you lay out the queue and the transforms on a canvas and decide where the quality gates go, and check your reasoning against the data pipeline interview questions interviewers ask about streaming. For the broker internals that follow a Kafka answer, such as ISR and consumer group rebalances or how transactions commit, work through the Kafka interview questions, and see how the round is run in the data engineering system design round guide.