Change data capture has settled into a few dominant patterns rather than continuing to reinvent itself: log-based capture over polling, an event stream (usually Kafka) as the transport layer between source and destination, and schema-aware handling of the inevitable upstream schema change. What's changed recently isn't the core mechanism — log-based CDC via tools like Debezium has been solid for years — it's where CDC gets used: increasingly as the backbone for keeping analytical warehouses fresh, not just for replicating between operational databases.
Log-based capture vs. polling
The older CDC pattern polls a source table on a timer, comparing rows against a previous snapshot or filtering on an updated_at column to find changes. This misses deletes (unless soft-deleted), misses intermediate states between polls, and adds read load to the source database proportional to poll frequency. Log-based CDC instead reads the database's own transaction log — the same log used for replication and crash recovery — capturing every insert, update, and delete as it happens, in order, without querying the source tables at all. This is why Debezium and similar tools are built around each database's native change stream (MySQL binlog, PostgreSQL logical replication slots, SQL Server CDC) rather than a generic polling query.
{
"before": { "id": 1042, "status": "pending", "total": 89.99 },
"after": { "id": 1042, "status": "shipped", "total": 89.99 },
"source": {
"table": "orders",
"ts_ms": 1753180800000,
"lsn": 24831902
},
"op": "u",
"ts_ms": 1753180800142
}Schema evolution is the recurring failure mode
The most common production incident with CDC pipelines isn't a dropped event — it's an upstream schema change breaking downstream consumers that assumed a fixed shape. Adding a nullable column is usually safe; renaming a column, changing a type, or dropping a column breaks anything downstream expecting the old shape. The pattern that survives this is treating the change stream as a versioned, schema-registered contract — using a schema registry (Confluent's, or an equivalent) so producers and consumers agree on compatible schema evolution rules, and consumers fail loudly on an incompatible change rather than silently ingesting corrupted data.
A schema registry with backward-compatibility enforcement catches breaking changes at publish time instead of at 3am when a downstream job starts throwing deserialization errors. Configure it to reject incompatible schema changes rather than treating compatibility as a manual review step someone might skip.
Exactly-once vs. at-least-once delivery
Most CDC pipelines guarantee at-least-once delivery, not exactly-once — a consumer can see the same change event twice after a rebalance or restart. This means downstream consumers need to be idempotent: applying the same update twice should produce the same result, typically by using the primary key plus the log sequence number or offset to deduplicate, or by using upserts (MERGE/INSERT ... ON CONFLICT) instead of blind inserts. Building a consumer that assumes exactly-once delivery is the second most common source of data-quality incidents after schema drift.
CDC as the warehouse freshness layer
The pattern gaining ground is using CDC to stream operational database changes directly into a warehouse's staging layer — replacing nightly batch extracts with near-real-time upserts — then letting existing batch transformation tools (dbt models running on a schedule) build downstream marts from that continuously-fresh staging data. This gets most of the value of real-time data (staging tables that are minutes, not a day, behind) without requiring every downstream transformation to become a streaming job.
CDC infrastructure has real operational cost — connectors to monitor, schema registries to maintain, consumer lag to watch. Reserve it for tables where freshness genuinely matters for a downstream decision. A slowly-changing reference table updated twice a year doesn't need a change stream; a nightly batch extract is simpler and just as correct.
Wrapping up
The CDC patterns that hold up in production are log-based capture (not polling), schema-registry-enforced compatibility on the change stream, and idempotent consumers designed for at-least-once delivery. Reach for CDC when a downstream system genuinely needs near-real-time freshness, and skip it — a scheduled batch extract is fine — when it doesn't.
Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.