I have spent a lot of time on high-volume transactional systems, including ones processing more than a million transactions a day. At that volume, every failure mode you think is rare happens daily. The one that hurts most is the duplicate: the customer charged twice, the order shipped twice, the refund issued twice.
Duplicates come from retries, and retries are correct behaviour. A client that times out cannot know whether the server finished. So the job is not to stop retries. It is to make them safe.
Three places duplicates come from
| Where | What happens | The fix |
|---|---|---|
| Client to API | A timeout hides a success; the client retries | Idempotency keys on every state-changing request |
| Database to events | The write commits but the event is lost, or the event goes out and the write rolls back | Transactional outbox |
| Events to consumers | The broker or relay redelivers after a failure | Idempotent consumers that track processed message IDs |
1. Idempotency keys at the edge
The client generates a unique key for each logical operation and sends it with the request. The server records the first outcome for that key and returns it for any retry. There is an IETF draft standardising this as an Idempotency-Key HTTP header, aimed at making non-idempotent methods like POST and PATCH fault-tolerant1.
Stripe's implementation is a good reference. It saves the status code and body of the first request for a key, including failures, and returns the saved result on retries. It also compares the parameters of a retry with the original and returns an error if they differ, which catches clients that reuse keys by mistake. Keys can be pruned once they are at least 24 hours old2.
Details I insist on:
- Keys are per operation, not per request. A retry reuses the key; a new purchase gets a new one.
- Store the key and the result in the same transaction as the business change, so they can't disagree.
- Handle concurrent duplicates. Two identical requests arriving together should not both proceed. A unique constraint on the key, or a short lock, settles it.
- Define the retention window and make clients aware of it.
2. The transactional outbox
A classic bug: the service writes the order to its database, then publishes an "order created" event. If it crashes between the two, the order exists but nothing downstream knows. Reverse the order and you can announce an order that was never saved.
The outbox pattern fixes this by writing the event into an outbox table in the same database transaction as the business data. A separate relay then publishes it, either by polling the table or by tailing the database log3. Change data capture tools are a common way to build the log-tailing relay4.
The outbox makes publishing reliable, not unique. A relay that restarts can publish the same event again, which is why the next step exists3.
3. Idempotent consumers
Every consumer should be safe to run twice on the same message. The simplest robust approach is to record processed message IDs in the consumer's own store, in the same transaction as its side effects, and skip any ID it has seen3.
Where you can, also make the operation naturally idempotent. "Set order status to shipped" is safe to repeat. "Add one to the shipped count" is not. Designing operations as state transitions rather than increments removes a whole class of bugs.
Data modelling that helps
- Unique constraints are your friend. One payment per idempotency key, one ledger entry per external transaction ID. Let the database refuse duplicates.
- Append-only ledgers. Record money movements as immutable entries and derive balances. Corrections are new entries, never edits.
- Explicit state machines for orders and payments, with allowed transitions. A retry that tries an invalid transition fails loudly.
- Reconciliation jobs that compare your records with the payment provider's every day. Something will drift eventually, and you want to find it before the customer does.
These choices work the same way in a relational database or in MongoDB with transactions and unique indexes. The discipline matters more than the engine.
A quick test for your system
- Kill the API process mid-request and retry. Does the customer get charged once?
- Kill the service between the database write and the event publish. Does downstream eventually see the event exactly once in effect?
- Redeliver yesterday's events to every consumer. Does anything change?
If any answer is "not sure", that is where I would start. In a technical due diligence I run exactly these questions, because they predict incidents better than any architecture diagram.
How I can help
I design and review payment, order and event-driven platforms for fintech and e-commerce teams, from idempotency and ledgers to scaling. As a fractional CTO I can also own the reliability roadmap. Or analyse your challenge and bring the result to a call.
- 1The Idempotency-Key HTTP Header Field (draft-ietf-httpapi-idempotency-key-header-07)IETF HTTPAPI Working Group · Oct 2025
- 2Idempotent requestsStripe API documentation
- 3Pattern: Transactional outboxmicroservices.io, Chris Richardson
- 4