- Client and provider retries made safe, so a timeout no longer risked a second charge
- Events published if and only if the payment write committed
- Reconciliation against providers automated, with exceptions routed to people
- A platform the in-house team could run and extend as volume grew past 1M+ transactions a day
Anonymised composite. Client names, identifying details and confidential figures are withheld; public facts are cited in the references.
The situation
A payments platform processing card and bank transfers for merchants was growing quickly. The architecture had served it well at lower volume: a payments service writing to Postgres, calling external providers, then publishing an event for the ledger and notifications. As traffic climbed towards and past 1M+ transactions a day, the edges started to fray. Mobile clients retried on timeouts. Providers occasionally responded slowly and then succeeded anyway. Events were sometimes published for writes that rolled back, or not published for writes that committed.
The symptoms were familiar: occasional duplicate charges, refunds issued by hand, and a reconciliation team matching provider settlement files to internal records in spreadsheets.
What had to be true
- Every step safe to retry, from the client to the provider call.
- One source of truth per payment, with state changes that cannot be lost or invented.
- Reliable events: published exactly when the database change commits.
- A ledger that balances, and reconciliation that runs itself for the normal cases.
- Operable by the in-house team, with clear signals when something drifts.
Architecture
The data flow, end to end
- KeyClient sends an Idempotency-Key; a repeat with the same key returns the stored result
- WritePayment row and outbox record written in one Postgres transaction
- CallProvider called with its own idempotency key derived from the payment ID
- PublishRelay reads committed outbox records from the database log and publishes to Kafka
- PostLedger consumes events idempotently and records balanced double entries
- ReconcileProvider settlement data matched to ledger entries; mismatches become exceptions
The edge followed the IETF Idempotency-Key header draft, which defines how clients make non-idempotent methods such as POST fault-tolerant1, and borrowed conventions familiar from established payment APIs, such as rejecting a reused key with different parameters2. The full pattern is in idempotent payments and order pipelines.
The hard part: the ambiguous middle
Duplicates at the edge are the easy case. The hard case is a request that times out somewhere in the middle: did the provider charge or not? The design handled it explicitly:
- The payment state machine had an "unknown" state for ambiguous provider outcomes, rather than guessing success or failure.
- Unknown payments were resolved by querying the provider, never by blindly retrying a charge.
- Provider calls carried their own idempotency keys, so a retry that did happen could not create a second charge on the provider's side.
- Events were produced through a transactional outbox: the change and the outbox record commit together and a relay publishes afterwards, which removes the dual-write problem without distributed transactions3. The relay used Debezium's outbox event router to turn outbox rows into routed Kafka events4.
Because the relay can deliver a message more than once, every consumer was idempotent too, keyed by event ID. CDC details are in change data capture and event streams.
Exactly-once is not a property you buy from a broker. It is what you get when every producer is reliable and every consumer is idempotent.
How the engagement ran
- Generates key
- Retries on timeout
- Gets the same result
- Deduplicates by key
- Writes payment and outbox
- Resolves unknown states
- Consumes idempotently
- Posts double entries
- Matches provider data
- Map failure modes
- Idempotency at the edge
- Tracing across hops
- Single write plus outbox
- CDC relay
- Idempotent consumers
- Double-entry postings
- Unknown-state handling
- Provider-side keys
- Automated matching
- Exception queue
- SLOs and runbooks
Every hop carried a trace context, so a single payment could be followed from the client request through the provider call, the outbox, the ledger posting and reconciliation when something looked wrong.
Service level objectives and alerting followed SLOs and error budgets for small teams.
Key decisions and trade-offs
| Decision | Alternative | Why |
|---|---|---|
| Idempotency keys at the API edge | Deduplicate downstream | Stops duplicates before any side effect happens |
| Transactional outbox with CDC | Write to database then publish | Events exist if and only if the write committed |
| Explicit unknown state | Treat timeouts as failures | Avoids both double charges and lost payments |
| Postgres as the payment source of truth | Event-sourced payments | Familiar to the team; strong transactional guarantees |
| Automated reconciliation with exceptions | Manual matching | Scales with volume; people handle only real anomalies |
What I'd do again, and differently
Again: name the ambiguous states in the data model and design for them explicitly. Differently: build reconciliation earlier. It is the system that tells you whether everything else is working, and it is the first place a new failure mode shows up.
If your platform is outgrowing its original design, the fractional CTO page describes how I work with teams on this, or analyse your challenge and bring the result to a call.
- 1The Idempotency-Key HTTP Header Field (Internet-Draft)IETF HTTPAPI Working Group
- 2Idempotent requestsStripe API documentation
- 3Pattern: Transactional outboxmicroservices.io, Chris Richardson
- 4Outbox Event RouterDebezium documentation