- Channels read stock from a continuously updated view instead of a nightly snapshot
- Order and stock events available to any new consumer without touching the source systems
- Replayable history, so projections can be rebuilt or corrected from the log
- Clear ownership of event contracts between the teams that produce and consume them
Anonymised composite. Client names, identifying details and confidential figures are withheld; public facts are cited in the references.
The situation
A retailer sold through physical stores, its own website and several marketplaces. Stock levels lived in a warehouse system and store systems, orders in an order management database, and each channel received a stock file once a night. During the day, the same last unit could be sold in a store, online and on a marketplace before anyone knew.
The workarounds made things worse: large safety buffers that hid sellable stock, manual cancellations, and ad hoc point-to-point integrations that each team had built to get fresher data for its own channel.
What had to be true
- Every committed change is captured, without polling or adding load to operational databases.
- No dual writes: an order and its event either both happen or neither does.
- Channels read a projection, not the source systems.
- History is replayable, so a bug in a projection can be fixed and rebuilt.
- Contracts are explicit, so source teams can change their schemas without breaking consumers.
Architecture
Log-based change data capture reads committed row-level changes from the database's transaction log, so applications consume changes rather than querying the source1. That became the backbone, with a stream processing layer building inventory views for each consumer.
The data flow, end to end
- CommitOrder or stock change and its outbox event written in one database transaction
- CaptureDebezium reads the log and routes outbox events to domain topics
- ValidateSchema checked against the registry; malformed events to dead-letter
- ProjectStream processor updates available-to-sell per SKU and location
- ServeRead model updated; channels query it or receive incremental updates
- ReconcilePeriodic comparison with source counts; drift raised, not hidden
More on the capture side in change data capture and event streams, and on duplicate-safe consumers in idempotent order pipelines.
The hard part: events that mean something
Streaming raw table changes was quick to demonstrate and the wrong long-term contract. Consumers ended up decoding internal columns and status codes, and any schema change in the source broke them. The transactional outbox pattern addresses this: the service writes a domain event to an outbox table in the same transaction as the business change, and a relay publishes it3. Debezium's outbox event router does that relay from the log, routing events to topics by aggregate type and keying them by aggregate identifier2.
- Events were designed with the consuming teams: "stock adjusted", "order allocated", "reservation released", not "row updated".
- Keys were chosen for ordering: everything about one SKU at one location lands in one partition, in order.
- Contracts went into the schema registry with compatibility rules, following a data contracts approach.
For current state, compacted topics keep the latest record per key and support rebuilding state after a failure4, which let new projections bootstrap without a bulk extract from the source.
CDC moves data quickly; it does not decide what the data means. Design the events with the teams who will read them.
How it ran
- Add outbox writes
- Own event schemas
- Keep compatibility
- Run Connect and Kafka
- Operate the registry
- Monitor lag and drift
- Read projections
- Report mismatches
- Retire old file feeds
- Stock and order flows
- Where overselling occurs
- Event catalogue draft
- One source with outbox
- One projection
- One channel in shadow
- Remaining sources
- Reservations logic
- Reconciliation jobs
- Channels cut over
- Nightly files retired
- Runbooks and on-call
Operationally, the signals that mattered were connector health, consumer lag per projection and the reconciliation drift report. Those went on one dashboard owned by the platform team, with alerts tied to what a channel would notice (stale stock) rather than to every broker metric. Traces carried the event identifier from commit to read model, so a disputed stock figure could be followed back to the change that produced it.
Key decisions and trade-offs
| Decision | Alternative | Why |
|---|---|---|
| Log-based CDC | Polling or nightly extracts | Captures every committed change with little source load |
| Transactional outbox | Raw table change events | Stable domain contracts; no dual writes |
| Key by SKU and location | Key by order | Ordered updates for the thing being counted |
| Separate read model | Channels query sources | Protects operational systems; fast reads |
| Reconciliation jobs | Trust the stream | Drift is surfaced and explained, not discovered by customers |
What I'd do again, and differently
Again: a thin slice end to end before scaling, shadow running against the old feed, and reconciliation from the start. Differently: agree the reservation and allocation rules with operations earlier. Live data exposed business rules that had never been written down, and those took longer to settle than the streaming platform.
If you are planning a similar move off batch, the fractional CTO page describes how I work with teams on it, or analyse your challenge and bring the result to a call.
- 1
- 2Outbox Event RouterDebezium
- 3Pattern: Transactional outboxmicroservices.io (Chris Richardson)
- 4Kafka design: log compactionApache Software Foundation