Design a Payment System: Idempotency, Ledgers and Reconciliation

A checkout page shows a spinner for eight seconds, then an error. The customer tries again and pays successfully. The next morning their bank statement shows two charges. Or the opposite: the provider charged the card, our server never heard back, the order was cancelled, and the money sits in limbo until an angry support ticket arrives a week later. Neither is a dramatic outage; both destroy trust, and both come from the same root cause: treating a payment like an ordinary API call.

A payment system is one of the few designs where correctness clearly beats speed. This case study designs the payment platform behind checkout and refunds for an online marketplace, using a payment service provider (PSP) to talk to card networks, UPI and wallets. The examples are hypothetical, and nothing here describes any particular company's internal systems.

• How to scope a payment platform and what its scale really is.
• Payment intents, attempts and an explicit state machine.
• Idempotency at each hop, and how to resolve unknown outcomes.
• Asynchronous confirmation with verified webhooks and status checks.
• A double-entry ledger, with a runnable example.
• Reconciliation, refunds, provider failover and the audit trail.
• Security: PCI scope, fraud and tampering.

The marketplace's order service needs to collect money from customers at checkout, refund them when orders are cancelled or returned, and know at all times how much is owed to each merchant and how much the platform has earned. Payments go through one or more external PSPs. The payment platform must never charge twice for one order, never lose track of money, and be able to explain every movement to finance teams and auditors.

Functional requirements: create a payment for an order; confirm it with a payment method; handle asynchronous outcomes; refund fully or partially; record every money movement; reconcile with the PSP; expose status to the order service.

Non-functional requirements

2 million payments/day -> about 23/s average; sale-day peaks of 5x -> about 120/sRefunds: 5% of payments -> 100,000/dayLedger: about 3 balanced entries per payment lifecycle (capture, provider fee, settlement)  plus refunds -> about 7 million entries/day, each with 2 to 4 linesStorage: 7 million entries x 500 bytes = 3.5 GB/day -> about 1.3 TB/year, kept for yearsPSP webhooks: about 2 to 3 events per payment -> about 6 million/day

Conclusions: the request rate is small; one well-run relational database (with a synchronous standby) can handle it, and gives us transactions, constraints and conditional updates, which are exactly what correctness needs. Storage grows steadily and is append-only, so partition ledger tables by time. Scale is not the hard part; failure handling is.

POST /v1/payment-intentsIdempotency-Key: checkout-ord_77{ "order_id": "ord_77", "amount": 149900, "currency": "INR", "customer_id": 42 }201 { "intent_id": "pi_51", "status": "requires_payment_method" }POST /v1/payment-intents/{intentId}/confirmIdempotency-Key: confirm-pi_51-1{ "payment_method_token": "pmt_..." }        (a PSP token from the PSP's client-side SDK or hosted page)202 { "status": "processing" }   or  200 { "status": "succeeded" }   or  { "status": "requires_action", "redirect_url": "..." }GET  /v1/payment-intents/{intentId}            -> current statusPOST /v1/refundsIdempotency-Key: refund-ord_77-item3{ "intent_id": "pi_51", "amount": 49900, "reason": "item_returned" }202 { "refund_id": "rf_9", "status": "pending" }POST /v1/psp/webhooks                          (PSP -> us; signature verified)

• The amount comes from the order service (server to server), never from the browser, so a customer cannot change what they pay.
• Every write takes an idempotency key, and the payment platform passes its own keys to the PSP.
• Outcomes are asynchronous by design: processing and requires_action are normal states, not errors.
• Internally, the platform emits events such as PaymentSucceeded, PaymentFailed and RefundSucceeded through an outbox (article 21), which the order service consumes.

A payment intent is "we want to collect ₹1,499 for order 77". An attempt is one try with one payment method at one PSP. A customer whose card is declined may try UPI next: one intent, two attempts.

CREATE TABLE payment_intents (    intent_id     VARCHAR(32) PRIMARY KEY,    order_id      VARCHAR(32) NOT NULL UNIQUE,     -- one intent per order    customer_id   BIGINT NOT NULL,    amount_paise  BIGINT NOT NULL CHECK (amount_paise > 0),    currency      CHAR(3) NOT NULL,    status        VARCHAR(24) NOT NULL,    version       BIGINT NOT NULL DEFAULT 0,    created_at    TIMESTAMPTZ NOT NULL DEFAULT now());CREATE TABLE payment_attempts (    attempt_id     VARCHAR(32) PRIMARY KEY,         -- also our idempotency key at the PSP    intent_id      VARCHAR(32) NOT NULL REFERENCES payment_intents(intent_id),    psp            VARCHAR(16) NOT NULL,    psp_reference  VARCHAR(64) UNIQUE,              -- the PSP's id for this attempt    status         VARCHAR(24) NOT NULL,    created_at     TIMESTAMPTZ NOT NULL DEFAULT now(),    updated_at     TIMESTAMPTZ NOT NULL);-- At most one successful attempt per intent:CREATE UNIQUE INDEX one_success_per_intent ON payment_attempts (intent_id) WHERE status = 'succeeded';CREATE TABLE refunds (refund_id PRIMARY KEY, intent_id, amount_paise, status, psp_reference UNIQUE, ...);CREATE TABLE psp_events (event_id VARCHAR(64) PRIMARY KEY, received_at, payload, processed_at);CREATE TABLE idempotency_keys (client_id, idem_key, request_hash, response, ... PRIMARY KEY (client_id, idem_key));

The constraints carry much of the correctness: one intent per order, one success per intent, one record per PSP reference, one processed row per webhook event.

requires_payment_method --confirm--> processing --PSP success--> succeeded --refund(s)--> partially_refunded / refunded        ^                               |  \--PSP failure--> failed (customer may try another method)        |                               |        +---- new attempt allowed ------+--requires 3-D Secure / app approval--> requires_action --> processing                                        \--no answer for a long time--> unknown -> resolved by status query

Every transition is a conditional update on the current state:

UPDATE payment_intents SET status = 'succeeded', version = version + 1WHERE intent_id = :id AND status IN ('processing', 'requires_action', 'unknown');

A duplicate or late event that tries to move a succeeded payment back to processing, or a refunded payment to succeeded, changes zero rows. That one rule protects against most webhook ordering problems (article 20).

Money movements are recorded in a double-entry ledger: every journal entry has lines that debit and credit accounts, and the debits and credits of each entry must balance. Entries are append-only; mistakes are corrected with new, reversing entries. This is the classic accounting structure, and it fits event sourcing naturally: the journal is the history, balances are derived (article 27; Microsoft's Event Sourcing guidance mentions ledgers among the good fits for the pattern (Event Sourcing pattern)).

journal_entries (entry_id PK, kind, reference (intent, refund or settlement id), created_at)journal_lines   (entry_id, line_no, account_id, amount_paise (debit positive, credit negative))accounts        (account_id, type: provider_clearing, merchant_payable:{merchant}, commission_revenue,                 provider_fees_expense, bank, ...)

Order service --charge(card details)--> PSP API, wait for the answer, store "paid" or "failed"

Simple, and wrong in several ways: card details pass through our servers (PCI scope), a timeout leaves us not knowing whether the card was charged, a retry may charge twice, asynchronous methods do not fit a single request, and there is no record of where money went.

1. Timeouts: the PSP call times out; the charge may or may not have happened.
2. Retries: clients, our services and networks retry; without idempotency, retries become duplicate charges.
3. Asynchronous outcomes: many methods complete minutes later; webhooks arrive late, twice, or out of order, or not at all.
4. Split brain between us and the PSP: our database says processing, the PSP says succeeded.
5. No money trail: without a ledger, finance cannot answer "how much do we owe merchant X?".
6. PSP outages during peak sales.

The AWS Builders' Library article on idempotent APIs describes the same discipline: unique caller-provided request identifiers, returning a semantically equivalent response for repeats, and rejecting reuse with different parameters (Making retries safe with idempotent APIs). Stripe's documentation describes how its idempotency keys work for retried requests (Idempotent requests).

1. Confirm request arrives (idempotency key checked).2. One transaction: create attempt (status processing), intent -> processing.3. Call the PSP with attempt_id as the idempotency key, with a timeout.4a. PSP answers succeeded/failed: one transaction updates attempt and intent (conditional),    writes ledger entries, writes outbox event; respond.4b. PSP answers requires_action: return the redirect or approval instructions; wait for a webhook.4c. PSP call times out: leave the attempt "processing" (outcome UNKNOWN, not failed); respond 202;    a resolver job queries the PSP for the attempt's status after a short delay.

The rule that prevents double charges: never treat a timeout as a failure and never start a new attempt while an earlier attempt's outcome is unknown. Resolve it first (by status query or webhook), then decide.

PSPs report asynchronous outcomes by calling a webhook endpoint. Stripe's documentation, for example, describes receiving events at an endpoint, verifying signatures, returning a success response quickly, and handling duplicate events (Webhooks). Our handler:

1. Verifies the signature (and timestamp, to resist replay).
2. Stores the raw event in psp_events keyed by event_id (a duplicate insert means "already have it": return 200).
3. Returns 200 immediately.
4. A processor applies the event: finds the attempt by psp_reference, applies the transition conditionally, writes ledger entries and the outbox event in the same transaction.
5. If the event refers to an unknown reference or an impossible transition, it is parked for investigation, not dropped.

Webhooks can be lost, so a resolver job also polls the PSP for any attempt that has stayed processing or requires_action beyond a threshold.

This Java program posts three balanced journal entries for one order (a ₹1,000 payment with a 2% marketplace commission, the PSP's processing fee, and a ₹500 partial refund), rejects an unbalanced entry, and derives account balances from the journal. The account names and fee are illustrative.

// File: LedgerDemo.javaimport java.util.ArrayList;import java.util.List;import java.util.Locale;import java.util.Map;import java.util.TreeMap;public class LedgerDemo {    // One line of a journal entry. Amounts are integers in paise; debits positive, credits negative.    record Line(String account, long amount) {}    record Entry(String id, String description, List<Line> lines) {}    static final List<Entry> journal = new ArrayList<>();   // append-only    static void post(String id, String description, Line... lines) {        long sum = 0;        for (Line line : lines) {            sum += line.amount();        }        if (sum != 0) {            throw new IllegalArgumentException(id + " rejected: debits and credits differ by " + sum);        }        journal.add(new Entry(id, description, List.of(lines)));    }    static Map<String, Long> balances() {        Map<String, Long> totals = new TreeMap<>();        for (Entry entry : journal) {            for (Line line : entry.lines()) {                totals.merge(line.account(), line.amount(), Long::sum);            }        }        return totals;    }    public static void main(String[] args) {        // A customer pays Rs 1,000 for an order; the platform keeps a 2% commission.        post("e1", "payment pay_51 captured",                new Line("provider_clearing", 100_000),        // money the provider owes us                new Line("merchant_payable", -98_000),         // what we owe the merchant                new Line("commission_revenue", -2_000));       // our earnings        // The provider charges us Rs 23.60 for processing, deducted at settlement.        post("e2", "provider fee for pay_51",                new Line("provider_fees_expense", 2_360),                new Line("provider_clearing", -2_360));        // Half the order is refunded: reverse half of the original split.        post("e3", "refund ref_9 on pay_51 (Rs 500)",                new Line("merchant_payable", 49_000),                new Line("commission_revenue", 1_000),                new Line("provider_clearing", -50_000));        try {            post("e4", "buggy entry", new Line("merchant_payable", 10_000), new Line("provider_clearing", -9_000));        } catch (IllegalArgumentException e) {            System.out.println(e.getMessage());        }        long total = 0;        for (Map.Entry<String, Long> b : balances().entrySet()) {            System.out.printf(Locale.US, "%-22s %10.2f%n", b.getKey(), b.getValue() / 100.0);            total += b.getValue();        }        System.out.println("sum of all balances: " + total + " (always zero in double-entry)");    }}

e4 rejected: debits and credits differ by 1000commission_revenue         -10.00merchant_payable          -490.00provider_clearing          476.40provider_fees_expense       23.60sum of all balances: 0 (always zero in double-entry)

Read the balances: the provider owes us ₹476.40 (₹1,000 collected, less ₹500 refunded, less its ₹23.60 fee); we owe the merchant ₹490; our commission is ₹10 after the refund; and the fee is an expense. Every rupee is accounted for, and the buggy entry could never enter the journal. In a real system the journal lives in database tables, each entry is written in the same transaction as the state change that caused it, and money is never stored as floating point.

Even with all of the above, our records and the PSP's can drift: a webhook lost and a resolver bug, a refund the PSP processed twice, a fee that changed. Reconciliation compares them independently, every day:

1. Download the PSP's settlement report (every transaction, fee and payout in a period).
2. Match each line to our attempts and refunds by psp_reference, comparing amounts and statuses.
3. Classify mismatches: present at the PSP but missing in our records, missing at the PSP, amount differs, status differs, fee differs.
4. Fix known patterns automatically (for example, apply a missed success through the normal transition path), and send the rest to a finance operations queue with full context.
5. Reconcile bank deposits against the PSP's payouts too.

A reconciliation that finds nothing for months is a good sign; one that is never run is a risk.

A refund is its own record with its own idempotency key and state machine (pending, succeeded, failed). Constraints ensure the total refunded never exceeds the captured amount (checked in the same transaction as creating the refund, with a row lock on the intent). The PSP call uses the refund_id as its idempotency key; outcomes arrive by response or webhook; ledger reversal entries are posted on success.

• Circuit breaker per PSP (article 23).
• Route new attempts to a secondary PSP when the primary is unavailable, if the business has integrated more than one.
• Never resend an in-flight attempt to a different PSP: if the first PSP's outcome is unknown, a second PSP could charge the same customer again. Resolve or expire the first attempt first.
• During a total PSP outage, the honest behaviour is to tell customers payments are temporarily unavailable, or to accept orders with "pay later" options if the business allows it.

Customer app --(card/UPI details)--> PSP's SDK or hosted page --> PSP   (card data never reaches us)Customer app --> Order service --(idempotent)--> Payment APIPayment API --> Payments DB (intents, attempts, refunds, idempotency keys, ledger, outbox)            --> PSP API (attempt_id as idempotency key; timeouts; circuit breaker)PSP --webhooks--> Webhook receiver (verify, store event, 200) --> Event processor (conditional transitions,                                                                   ledger entries, outbox)Resolver job --> PSP status API for stuck attemptsOutbox relay --> PaymentSucceeded / PaymentFailed / RefundSucceeded --> Order service, notificationsReconciliation (daily) --> PSP settlement reports vs ledger --> auto-fix or finance ops queuePayments DB: primary + synchronous standby; point-in-time backups; ledger tables partitioned by month

• Inside the payments database: strong consistency. State transitions, ledger entries and outbox events commit together, so the payment state and the money trail never disagree.
• With the PSP: eventual. Our records converge to the PSP's truth through responses, webhooks, status queries and reconciliation.
• With the order service: eventual, through events and the order saga (article 21), with explicit intermediate states (payment_pending).
• No floating point, no edits: money is integer paise; history is append-only.

• Keep card data out: use the PSP's client-side SDK or hosted page so card details go straight to the PSP, and store only tokens and references. PCI DSS applies to entities that store, process or transmit cardholder data (PCI DSS); keeping that data out of your systems greatly reduces what falls in scope.
• Verify every webhook signature and timestamp; accept webhooks only on a dedicated endpoint.
• Server-side amounts: the amount is computed from the order on the server.
• Least privilege: only the payment service can write payment tables; finance tools read through audited paths; refunds above a threshold need approval.
• Secrets: PSP keys in a secrets manager, rotated, with separate keys per environment.
• Fraud: velocity limits (attempts per card, account and device), the PSP's risk signals, 3-D Secure or equivalent customer authentication where required, and review queues for suspicious refunds.
• Audit: every state change, refund and manual adjustment records who, what and when, in an append-only audit log.

"Payments here are about correctness, not scale: about 2 million payments a day, 120 a second at peak, on one relational database with a synchronous standby, because transactions and constraints are what keep money safe.

Each order gets one payment intent; each try with a method is an attempt. Card and UPI details go straight to the PSP through its SDK, so we only hold tokens. Every hop is idempotent: the order service calls us with a key per order, we call the PSP with the attempt id as its idempotency key, and webhooks are deduplicated by event id. A timeout is an unknown outcome, never a failure: the attempt stays processing until a resolver job or a webhook tells us the truth, and we never start a new attempt while one is unknown.

State changes are conditional updates on an explicit state machine, so late or duplicate events can't move a payment backwards. Each state change commits with double-entry ledger entries and an outbox event in one transaction, so the order service learns the outcome reliably and every rupee is accounted for. Refunds work the same way. Every day we reconcile the ledger against the PSP's settlement reports and investigate mismatches. Card data never enters our systems, webhooks are signature-verified, amounts are computed server-side and everything is audited."

e1 payment captured:  provider_clearing +2,000 ; merchant_payable -1,900 ; commission_revenue -100e2 provider fee:      provider_fees_expense +40 ; provider_clearing -40e3 full refund:       merchant_payable +1,900 ; commission_revenue +100 ; provider_clearing -2,000

• Next: Design a Video Streaming Platform: Upload, Transcoding and Adaptive Streaming starts the Level 3 case studies.
• Later: the food delivery (42) and e-commerce (43) designs use this payment platform inside larger sagas.