Kafka vs Kinesis, from someone who has paged for both

Reality: Provisioned mode is manual sharding; you split and merge shards explicitly, and resharding invalidates KCL checkpoints in the affected hash range. On-Demand auto-scales but ramps slowly (roughly doubling every 15 minutes), so a synthetic 10x spike will throttle. Either way you own a runbook.

Reality: only past roughly 50 MB/sec sustained. Below that, the three-broker minimum and the engineer-hour tax on self-hosted Kafka make Kinesis cheaper TCO. The cost crossover is real and belongs in your interview answer when someone says 'cheaper' without naming a scale.

Reality: AWS manages broker hosts, patching, and AZ-aware placement. AWS does not manage your topics, partitions, consumer groups, ACLs, MirrorMaker, Schema Registry, or Connect clusters. The delta vs self-hosted is real but smaller than the marketing suggests. MSK Serverless absorbs broker capacity, which is meaningful.

Reality: Kafka exactly-once is producer-side (idempotent producers + transactions + read-committed consumers). Kinesis exactly-once is consumer-side (KCL checkpoints sequence numbers; the application keeps side-effects idempotent). Different mechanics, different failure modes. Saying 'exactly-once' without naming the side is an L4 ceiling.

Reality: Firehose buffers by size or time and writes batched objects. Within an object, records preserve arrival order from Kinesis; across objects, ordering is best-effort. Firehose retries on failure and can produce duplicate objects on retry boundaries. Real exactly-once requires deduplication on the consumer side keyed off the Kinesis sequence number or producer-supplied id.

On pre-KRaft clusters. One AZ blips, the ZK ensemble loses majority, Kafka brokers survive but cannot accept metadata changes. Producers keep writing for a while, then start failing on leader-not-available. Fix: KRaft. If you can't move yet, run ZK across three AZs with its own observability and treat ZK incidents as P1 even when Kafka looks fine.

A pod restart loop in a stateful consumer group triggers eager rebalances every cycle; the group stops processing each time. Fix: cooperative-sticky assignor (KIP-429), longer session.timeout.ms, and stop deploying with rolling restarts that bounce too fast.

Replicas fall behind because broker disk I/O is saturated; ISR drops to 1; min.insync.replicas=2 with acks=all blocks producers. The page is 'producers are stuck'; the cause is replication fetcher lag. Fix: bigger EBS IOPS, segment.bytes tuning, or fewer partitions per broker.

On compacted topics with high write rate. The cleaner can't keep up; your changelog topic grows unbounded. Fix: log.cleaner.threads, log.cleaner.io.max.bytes.per.second, and accepting that compacted topics are not free.

The new controller has to refresh metadata for every partition; producers see leader-not-available for the duration. On a 5K-partition cluster, that's minutes of degraded write availability. Mitigation: smaller clusters, or KRaft (faster controller failover).

The producer hits the per-shard 1 MB/sec or 1000 records/sec cap. The math was right on average but a hot key (a single high-traffic user_id) lands all writes on one shard. Fix: salt the partition key with a hash mod-N suffix, re-aggregate downstream. On-Demand mode doesn't save you; it scales total capacity, not per-key fanout.

The 5 GetRecords/sec/shard ceiling kicks in when too many consumers share a shard. Fix: Enhanced Fan-Out (one HTTP/2 stream per consumer per shard, billed separately) or fewer consumers.

A consumer holds a shard iterator longer than 5 minutes. KCL consumers that lag past their lease get this and have to recover. Fix: faster polling, or accept the iterator refresh churn.

You split a hot shard to recover from throttling; KCL leases for the parent shard are invalidated for the affected hash range; the application restarts processing from a new sequence number. If your downstream isn't idempotent, you double-process. Plan for this; don't discover it during the page.

A promo launch 10x your traffic in 30 seconds. On-Demand ramps over roughly 15 minutes per doubling, so the first few minutes throttle. Fix: pre-warm by running synthetic load, or use Provisioned with explicit pre-splits ahead of the launch.

Each producer gets a producer ID and sequence numbers per partition; the broker dedupes on retry within the same session. Default since 3.0.

Wrap multi-partition writes (and consumed offsets) in an atomic commit via the transaction coordinator. A consumer reading with isolation.level=read_committed only sees committed data, never aborted batches.

The Kafka Streams pattern commits input offsets and output records in the same transaction, so a failure between reading and writing is rolled back atomically.

Latency from coordinator round-trips, plus the operational weight of running a transaction coordinator. Worth it for stream processing; overkill for fire-and-forget event ingest.

Stores the last-processed sequence number per shard lease in DynamoDB. On restart, the consumer resumes from the checkpoint, not from where it died.

Your responsibility. KCL guarantees at-least-once; exactly-once requires the application to dedupe on sequence number or a producer-supplied id.

When you split or merge shards, the parent shard lease ends and KCL takes leases on the children. The checkpoint boundary is preserved, but the application has to handle the new sequence number space cleanly.

If you don't want to manage KCL state, a Flink job with the Kinesis source connector and exactly-once sinks moves the guarantee back into the framework.