Debezium is the de facto open-source standard for log-based CDC, distributed as a set of Kafka Connect source connectors — one per supported database (MySQL, PostgreSQL, SQL Server, MongoDB, Oracle, and others) — that read each database's native change stream and publish structured change events to Kafka topics. It's not a standalone service you run by itself; it runs inside Kafka Connect, which handles the connector lifecycle, offset tracking, and scaling.
How the connector reads changes
For PostgreSQL, Debezium uses logical replication slots and the write-ahead log (WAL) to stream every committed change in order. For MySQL, it reads the binary log (binlog) the same way a MySQL replica would. In both cases, the mechanism is the database's own built-in replication protocol — Debezium is acting as a replica that, instead of applying changes to another database, serializes them as events and publishes them to Kafka. This is why enabling CDC on a source database has real prerequisites: PostgreSQL needs wal_level = logical and a replication slot; MySQL needs binlog enabled in row format.
{
"name": "orders-connector",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"database.hostname": "prod-db.internal",
"database.port": "5432",
"database.user": "debezium_replicator",
"database.dbname": "orders_db",
"topic.prefix": "orders",
"table.include.list": "public.orders,public.order_items",
"plugin.name": "pgoutput",
"slot.name": "debezium_orders_slot",
"publication.autocreate.mode": "filtered"
}
}The initial snapshot
A change stream only captures changes going forward — it has no history of rows that existed before the connector started. Debezium solves this with a snapshot phase: on first startup, it takes a consistent snapshot of the existing table data (using the database's own consistent-read mechanisms so it doesn't miss or duplicate rows relative to the point where streaming begins), publishes that as a set of "read" events, then transitions seamlessly into streaming live changes from the log. Understanding this two-phase behavior matters operationally — a large initial snapshot on a multi-billion-row table can take hours and adds read load to the source database, so it's usually scheduled for low-traffic windows.
Debezium supports snapshot modes beyond the default — never skips the snapshot entirely and starts from the current log position (useful when you only care about changes from now on), and initial_only takes the snapshot and stops without streaming. Pick the mode that matches whether downstream consumers need full history or just changes going forward.
Single Message Transforms and topic routing
Kafka Connect's Single Message Transform (SMT) framework lets you reshape Debezium's verbose change-event envelope — which includes the full before/after row state plus source metadata — before it lands in Kafka. The ExtractNewRecordState SMT is the most commonly applied one: it flattens the envelope down to just the "after" state (or a tombstone for deletes), which is what most downstream consumers actually want rather than the full CDC envelope.
A PostgreSQL replication slot holds WAL segments on disk until Debezium consumes them. If the Debezium connector goes down and nobody notices, the WAL keeps accumulating on the source database and can eventually fill the disk, taking the production database down with it. Monitor replication slot lag as a first-class production metric, not an afterthought.
Delete events and tombstones
A row delete produces a change event with a null "after" state, followed (by default) by a Kafka tombstone record — a message with the same key and a null value — which signals to Kafka's log compaction that all prior records for that key can eventually be removed. Consumers that build a materialized view from the change stream need to handle both: the delete event to know a row was removed, and awareness that the tombstone exists for compaction, not as a second business event to process.
Wrapping up
Debezium's core value is turning each database's native replication log into structured, ordered Kafka events without querying the source tables directly. Enable logical replication or binlog capture ahead of time, plan for the initial snapshot's load on the source database, flatten the event envelope with an SMT for downstream consumers, and monitor replication slot lag as a production-critical metric — an unconsumed slot is a disk-filling incident waiting to happen.
Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.