The hardest failure is an ambiguous success
Payment systems are full of timeouts. A client sends a request to create a transfer. The provider accepts it. The network drops before the response reaches your backend. From the caller’s perspective the request failed. From the provider’s perspective the transfer may already exist. If the application blindly tries again, it may create a second transfer.
This is why the most dangerous payment bug is often not a clean failure. Clean failures are easier to reason about. The user sees an error, the system records a rejection, and no money moves. Ambiguous success sits between states. One component knows the operation succeeded while another does not.
Retries are unavoidable. Mobile clients retry. Queues retry. HTTP libraries retry. Operations staff retry. Providers redeliver webhooks. A reliable design therefore asks a different question: what happens when the exact same logical instruction arrives more than once?
If the answer is “we hope it does not”, the system is waiting for a duplicate.
An idempotency key is a business contract
Idempotency means repeating the same operation produces the same effective result rather than creating a second consequence. Stripe documents this explicitly for API writes: a client supplies an idempotency key, and repeated requests using the same key can return the original result instead of performing the operation again.
The important part is not the header name. It is the scope of the logical operation. If a user presses “Send ₦50,000 to beneficiary B” once, every retry generated by that action should carry the same operation identifier. If the user later intentionally creates a second transfer for the same amount and beneficiary, that is a new logical operation and needs a new key.
The server should bind the key to the normalized request parameters. Reusing the same key for different instructions should fail rather than silently returning the first result. The record should also have a lifecycle appropriate to the domain. A financial platform may need to preserve durable operation references well beyond the short retention period used by an external API because its own audit and reconciliation requirements are different.
Idempotency works best when it is part of the domain model, not middleware sprinkled around selected endpoints.
The database should make duplicates structurally difficult
Application checks are not enough. Two requests can both execute “does this key exist?” before either inserts the row. If the database allows both inserts, the race has already won. A unique constraint on the idempotency key or provider reference turns the invariant into something the database can enforce under concurrency.
The write path should also keep the idempotency record and the resulting business mutation in one transaction where possible. Imagine recording the payment first and the idempotency key second. The process crashes between those steps. A retry arrives, sees no idempotency record and creates another payment. Reversing the order can create the opposite problem: a key exists for a payment that never committed. Atomicity matters.
PostgreSQL unique constraints and transactional semantics are useful here because correctness does not depend on every application instance seeing the same memory state. The database becomes the final arbiter of whether the operation identifier has already been consumed.
For provider calls that cannot participate in the same database transaction, the state machine needs explicit intermediate states such as created, submitted, provider-unknown, provider-confirmed, failed and reversed. Pretending the network call is atomic with your database write is how ambiguous states disappear from the code while remaining very real in production.
Concurrency turns a simple retry into a race condition
Duplicate requests do not always arrive one after another. A user can double-tap. Two app servers can process the same queue message. A webhook can arrive while a polling job is already reconciling the transaction. A timeout handler can retry before the first provider request finishes.
The design must therefore be safe under simultaneous execution. Depending on the operation, that can mean unique constraints, row locks, compare-and-swap versions, serializable transactions or a single-writer queue for a particular resource. The right choice depends on throughput and contention, but the invariant should be explicit.
Consider wallet debit. “Balance is sufficient” and “subtract amount” must not be independent statements executed without concurrency control. Two workers can both observe the same balance and both spend it. The same principle applies to one-time coupon use, inventory reservation, payout release and approval transitions. Financial bugs are often ordinary race conditions with extraordinary consequences.
Webhook consumers should assume duplicate delivery
Webhooks are notifications, not magic exactly-once messages. Providers retry when endpoints time out or return errors. Networks duplicate and reorder delivery. Your own handler may crash after committing the business change but before returning a successful HTTP response. The provider then sends the event again.
Webhook processing should therefore verify authenticity, persist the event identity, acknowledge promptly and move heavy work to a controlled asynchronous path. Before applying a side effect, the consumer checks whether the provider event or underlying resource transition has already been processed. The Stripe Developers video embedded below specifically covers verification, duplicate-event handling, idempotency and quick acknowledgement.
It is also useful to distinguish event identity from business identity. Two different webhook event IDs can describe the same underlying payment state change in some integration models. The consumer may need to deduplicate at both layers: do not process the exact event twice, and do not post the same business consequence twice.
A ledger should record the consequence once
If money moves inside the platform, the ledger is where duplicate effects become visible. The transaction service may call a provider once or twice, but the internal accounting consequence must be deterministic. A successful external transfer should map to a defined set of ledger entries with a unique business reference. Reprocessing the same provider result should find those entries rather than posting a second set.
This is one reason balances should not be treated as a mutable number that handlers increment and decrement without history. Durable entries make it possible to trace the source operation, reverse it correctly and reconcile the platform against the provider. If the ledger has a unique constraint on the external or internal transaction reference, it becomes another barrier against duplication.
Idempotency does not eliminate the need for reversals. A provider can legitimately settle a transaction and later reverse it. That is a new business event, not a duplicate of the first one, and should create compensating ledger entries rather than rewriting the original.
Reconciliation is how the system admits the network can lie by omission
No matter how careful the request path is, external systems can leave you uncertain. A provider request times out. A webhook never arrives. A status endpoint temporarily disagrees with the callback. Reconciliation closes that gap by comparing internal records with authoritative external state.
Transactions in “unknown” or “submitted” states should be revisited by a bounded worker. The worker uses provider identifiers, client references or idempotency keys to ask what actually happened. It then advances the internal state through an authorized, idempotent transition. The job itself must be safe to rerun.
Operational dashboards should surface unresolved ambiguity. A transaction that has been “processing” for six hours is not merely a UI problem. It may represent money that moved externally without an internal ledger post, or an internal reservation that should have been released. Good reconciliation turns those mismatches into work queues before customers discover them.
A safer payment write path
A practical payment mutation can be summarized as a sequence of invariants. Generate or accept a stable operation ID. Authenticate the actor and authorize the specific account or wallet. Validate limits and current business state. Create the local transaction record transactionally with the idempotency key. Submit to the provider with a stable provider-facing reference. Record the provider result without assuming a timeout means failure. Post ledger consequences exactly once. Consume webhooks idempotently. Reconcile anything left ambiguous.
Then test the ugly paths. Send the same request concurrently. Drop the connection after the provider receives it. Deliver the same webhook five times. Deliver events out of order. Crash the worker after the database commit but before acknowledgement. Retry after a deployment. Force a unique-constraint collision. A payment system is ready when these tests produce boring outcomes.
The goal is not to prevent retries. Retries are how distributed systems recover. The goal is to make retries safe enough that recovery does not become a second payment.
Sources and further reading
