Kafka Connect vs Debezium

Kafka Connect moves data between Kafka and other systems through reusable connectors, so each integration is configuration instead of a bespoke consumer or producer. The Kafka Connect documentation describes 2 modes. In standalone mode all work runs in a single process, which is simple but has no fault tolerance. In distributed mode workers balance connectors and tasks among themselves, and you can scale the group up or down. Configs and offsets live in Kafka topics, and statuses are stored there too. Connectors are managed through a REST API that listens on port 8083 by default.

Connectors can apply transformations, which the documentation limits to lightweight message-at-a-time changes. A sink connector can route records it fails to process to a dead letter queue topic, set with errors.deadletterqueue.topic.name. None of this is specific to CDC: the same runtime hosts connectors that write Kafka topics to object storage, or to search indexes and warehouses.

Kafka Connect also offers exactly-once delivery for source connectors, since Apache Kafka 3.3.0 and in distributed mode only, switched on with the worker setting exactly.once.source.support. A connector has to support it, so check the connector before counting on it.

A polling connector runs a query on a schedule, for example every row whose updated_at is newer than the last value it saw. It can only report rows that exist when the query runs. A row deleted between polls is invisible, 2 updates between polls collapse into 1, and the query adds load to the source database on every cycle.

Debezium reads the transaction log instead. For PostgreSQL that is logical decoding through a replication slot, with the pgoutput plug-in that ships with PostgreSQL 10 and later. Every committed change becomes an event with a key and the row before and after the change. Each event also has an op code and a source block that carries the log position. Updates and deletes arrive as they happened, in log order, with no query against your tables.

The first start takes a snapshot so consumers begin with the current rows. The PostgreSQL connector's default snapshot mode is initial: it snapshots when no stored offset exists, then streams from the recorded log position. Downstream, the ExtractNewRecordState transform flattens the envelope to just the new row for sinks that expect plain records, and a delete produces a delete event followed by a tombstone record that lets Kafka log compaction drop the key.

Debezium Server runs 1 source connector per instance and streams its events directly to targets such as Amazon Kinesis, Google Cloud Pub/Sub, Apache Pulsar, Redis Streams, RabbitMQ and Kafka itself, all of which the documentation lists. It takes the offset storage that Connect would otherwise provide and defaults to a file, with in-memory and JDBC alternatives, and Redis too.

There is no Connect REST API, no group of workers rebalancing tasks and no other connectors in the same runtime. Connectors that keep a schema history, which includes MySQL, Oracle, SQL Server and Db2, default to storing it in Kafka, so a deployment without Kafka must configure another store. The embedded engine goes further and puts the connector inside your own Java process, which fits when change events feed 1 application and a separate runtime is overhead.

Log-based capture moves cost from query load to log retention. On PostgreSQL, the replication slot is guaranteed to retain all WAL entries Debezium still needs, even during a connector outage, according to the Debezium PostgreSQL connector documentation. A connector that stays down while the source keeps writing makes the database hold more and more WAL on its own disk.

Watch 2 columns of pg_replication_slots: confirmed_flush_lsn, how far the connector has received, and restart_lsn, how far the database can reclaim. If the first keeps advancing while the second lags, the database isn't reclaiming space. A table that rarely changes can cause this, and the documented remedy is periodic heartbeat events through heartbeat.interval.ms.

Plan for duplicates as well. The Debezium FAQ says consumers see each event once in normal operation, and at least once after a crash. Make sinks idempotent by deduplicating on the primary key and the log position in the event's source block, and treat Connect's exactly-once setting as a help for the connector side, not a guarantee for the whole pipeline.

Debezium on Kafka Connect fits when you already run Kafka, need every change in order with deletes included, or want several sources and sinks managed in 1 runtime with a REST API. The cost is a Connect cluster and a log-retention habit on each source database.

A polling connector fits when the table has a reliable updated_at and deletes are soft or irrelevant. It also fits when you can't enable logical replication on the source or an approximate copy refreshed every few minutes is enough. It is simpler to grant access for, and it gives up deletes and intermediate states.

Debezium Server fits when the destination is Kinesis or a comparable service such as Pub/Sub, and standing up Kafka only to carry change events is not worth it. The embedded engine fits when 1 application consumes the changes itself. Each of these trades the Connect runtime for less infrastructure and fewer features, so pick by what your destination and your team already run.

If the choice comes up in an interview, it is usually inside a design question: how do you keep a warehouse or search index in sync with an operational database. The CDC pipeline interview questions cover that ground, and the system design round guide shows how to structure the answer. For the batch side of the same tradeoff, see batch vs streaming pipelines.

Interviewers ask about change data capture to see whether you understand how ordering and deletes behave and how snapshots and replay work, not whether you know tool names. Work through the data pipeline interview questions and the data pipeline practice problems, and compare the transport choices in Kafka vs Kinesis.