Kafka vs Kinesis, from someone who has paged for both

Five failure modes you have to know cold to defend a Kafka design.

• ▸ZooKeeper quorum loss on pre-KRaft clusters. One AZ blips, the ZK ensemble loses majority, Kafka brokers survive but cannot accept metadata changes. Producers keep writing for a while, then start failing on leader-not-available. Fix: KRaft. If you cannot move yet, run ZK across three AZs with its own observability and treat ZK incidents as P1 even when Kafka looks fine.
• ▸Rebalance storms from consumer churn. A pod restart loop in a stateful consumer group triggers eager rebalances every cycle; the group stops processing each time. Fix: cooperative-sticky assignor (KIP-429), longer session.timeout.ms, and stop deploying with rolling restarts that bounce too fast.
• ▸ISR shrinkage under load. Replicas fall behind because broker disk I/O is saturated; ISR drops to 1; min.insync.replicas=2 with acks=all blocks producers. The page is “producers are stuck”; the cause is replication fetcher lag. Fix: bigger EBS IOPS, segment.bytes tuning, or fewer partitions per broker.
• ▸Log compaction lag on compacted topics with high write rate. The cleaner cannot keep up; your changelog topic grows unbounded. Fix: log.cleaner.threads, log.cleaner.io.max.bytes.per.second, and accepting that compacted topics are not free.
• ▸Controller failover after a broker JVM pause. The new controller has to refresh metadata for every partition; producers see leader-not-available for the duration. On a 5K-partition cluster, that is minutes of degraded write availability. Mitigation: smaller clusters, or KRaft (faster controller failover).

Five throttling and lease pathologies that page Kinesis-on-call.

• ▸PROVISIONED_THROUGHPUT_EXCEEDED on writes. The producer hits the per-shard 1 MB/sec or 1000 records/ sec cap. The math was right on average but a hot key (a single high-traffic user_id) lands all writes on one shard. Fix: salt the partition key with a hash mod-N suffix, re-aggregate downstream. On-Demand mode does not save you; it scales total capacity, not per-key fanout.
• ▸GetRecords throttling on reads. The 5 GetRecords/sec/shard ceiling kicks in when too many consumers share a shard. Fix: Enhanced Fan-Out (one HTTP/2 stream per consumer per shard, billed separately) or fewer consumers.
• ▸Stale iterator exception. A consumer holds a shard iterator longer than 5 minutes. KCL consumers that lag past their lease get this and have to recover. Fix: faster polling, or accept the iterator refresh churn.
• ▸Resharding mid-incident. You split a hot shard to recover from throttling; KCL leases for the parent shard are invalidated for the affected hash range; the application restarts processing from a new sequence number. If your downstream is not idempotent, you double-process. Plan for this; do not discover it during the page.
• ▸On-Demand ramp lag. A promo launch 10x your traffic in 30 seconds. On-Demand mode ramps over roughly 15 minutes per doubling, so the first few minutes throttle. Fix: pre-warm by running synthetic load, or use Provisioned with explicit pre-splits ahead of the launch.

• ▸Idempotent producer (enable.idempotence=true). Each producer gets a producer ID and sequence numbers per partition; the broker dedupes on retry within the same session. Default since 3.0.
• ▸Transactions wrap multi-partition writes (and consumed offsets) in an atomic commit via the transaction coordinator. A consumer reading with isolation.level=read_committed only sees committed data, never aborted batches.
• ▸Read-process-write loops (the Kafka Streams pattern) commit input offsets and output records in the same transaction, so a failure between reading and writing is rolled back atomically.
• ▸Cost: latency from coordinator round-trips, plus the operational weight of running a transaction coordinator. Worth it for stream processing; overkill for fire-and-forget event ingest.

• ▸KCL checkpointing stores the last-processed sequence number per shard lease in DynamoDB. On restart, the consumer resumes from the checkpoint, not from where it died.
• ▸Idempotent sinks are your responsibility. KCL guarantees at-least-once; exactly-once requires the application to dedupe on sequence number or a producer-supplied id.
• ▸Resharding interaction. When you split or merge shards, the parent shard lease ends and KCL takes leases on the children. The checkpoint boundary is preserved, but the application has to handle the new sequence number space cleanly.
• ▸Flink as the alternative. If you do not want to manage KCL state, a Flink job with the Kinesis source connector and exactly-once sinks moves the guarantee back into the framework.