When I re-architect a system around events, the first question is usually "do we have to change every service to publish events?". Often the answer is no. Every write already goes through the database, and the database already keeps an ordered log of every change. Change data capture reads that log and turns it into a stream.
Polling versus the log
The simple approach is polling: query for rows with a newer updated_at every few seconds. It misses deletes, adds load to the database, and struggles with ordering. Log-based CDC reads the database's own change log instead. Debezium's documentation describes it as capturing all data changes with very low delay, in the millisecond range for databases such as MySQL and PostgreSQL, without the CPU cost of frequent polling2.
| Source | Native mechanism | Common tooling |
|---|---|---|
| PostgreSQL | Logical decoding of the write-ahead log4 | Debezium connector |
| MongoDB | Change streams over the oplog3 | Change streams directly, or Debezium |
| MySQL and others | Binary log | Debezium connectors for Kafka Connect1 |
Where CDC earns its keep
- Feeding analytics and search without nightly batch jobs: the warehouse, search index and vector index stay seconds behind the source.
- Keeping RAG indexes fresh. When a document or record changes, the change event triggers re-embedding, so retrieval never serves stale content. This pairs naturally with hybrid search in MongoDB.
- Strangling a monolith. New services subscribe to changes in the legacy database while it keeps running.
- Audit and lineage, because every change becomes an immutable event with a position in the log.
Raw tables or deliberate events?
The fastest way to start is to stream tables as they are. The cost is coupling: every consumer now depends on your internal schema, and renaming a column becomes a breaking change across teams.
For events other teams depend on, I combine CDC with the transactional outbox pattern. The service writes a deliberate, versioned event into an outbox table in the same transaction as its business data, and CDC publishes the outbox5. You get the reliability of the log with a contract you control. I cover the pattern for payments and orders in idempotent payment and order pipelines.
Raw table streams for internal replication and analytics; outbox events for anything another team builds on. Mixing the two up is a common CDC design mistake.
Pitfalls to plan for
- Initial snapshots. Connectors usually begin by snapshotting current state before streaming changes. Debezium supports initial and incremental snapshots2; plan for the load and the time on large tables.
- Log retention. If a connector is down longer than the database keeps its log, you lose your place and need a new snapshot. Monitor connector lag, and size retention for your worst outage.
- Schema changes. Decide how consumers handle new and removed fields before the first migration, not during it.
- Sensitive data. Change events carry whatever the row carries. Filter columns and mask values at the connector, which Debezium supports2, rather than trusting every consumer.
- Duplicates. After restarts, events can be redelivered. Consumers must be idempotent.
A sensible first step
Pick one high-value flow, such as orders into the warehouse or documents into a search index. Stream it with CDC, measure end-to-end lag, and run it alongside the existing batch job until the numbers match. Then retire the batch job. It is the lowest-risk path from nightly ETL to real time, and it is a pattern I have used on high-volume systems.
How I can help
I design event-driven data platforms, from CDC and outbox patterns to streaming into warehouses, search and AI retrieval. As a fractional CTO I can own the migration plan. Or analyse your challenge and bring the result to a call.
- 1Debezium documentationDebezium
- 2Debezium featuresDebezium
- 3Change StreamsMongoDB documentation
- 4Logical DecodingPostgreSQL documentation
- 5Pattern: Transactional outboxmicroservices.io, Chris Richardson