Idempotency, Retries and Message Delivery Guarantees

A customer taps "Pay ₹499" on a phone with a weak signal. The request reaches the server, the card is charged, and the response is lost in a tunnel. The app shows "Something went wrong, try again". The customer taps again. Without protection, they are charged ₹998, and your support team spends the afternoon issuing refunds.

Article 18 explained why this happens: a timeout cannot tell success from failure, so retries are unavoidable, and retries create duplicates. This article is about making duplicates harmless. It is one of the most practical topics in the course and one of the most common SDE-2 interview probes: "what happens if this request is retried?"

• What idempotency means for APIs, messages and database writes.
• Every common source of duplicate requests.
• How to implement idempotency keys correctly, including concurrency and parameter checks.
• Unique constraints and natural keys as deduplication.
• Idempotent consumers for queues and event streams.
• Which failures to retry, how often, and with what delays.
• Exponential backoff, jitter, retry budgets and retry amplification.
• What at-most-once, at-least-once and "exactly-once" really mean.

An operation is idempotent if performing it several times has the same effect as performing it once.

For APIs, a useful definition has two parts: a repeated request has the same effect, and ideally gets the same response, so a client that retries cannot tell whether its first attempt arrived. AWS describes exactly this goal in its Builders' Library article on idempotent APIs: return a semantically equivalent response for every request carrying the same unique client request identifier, so retries are invisible to calling code (Making retries safe with idempotent APIs).

Two ways to get there:

1. Design operations to be naturally idempotent: express them as "set to this state" instead of "change by this amount", and use conditional transitions (article 14).
2. Deduplicate operations that cannot be made naturally idempotent (creating, charging, sending): recognise a repeat and do not act again.

The lesson: duplicates are not an edge case. In any system that retries, they are routine.

An idempotency key is a unique identifier the client generates for one logical operation (for example a random UUID created when the user opens the checkout screen) and sends with every attempt of that operation, commonly in an Idempotency-Key header. Payment providers popularised the pattern: Stripe's documentation describes saving the status code and body of the first request made with a key, returning the same result for later requests with that key (including errors), comparing parameters to catch misuse, and allowing keys to be pruned after they are at least 24 hours old (Idempotent requests). An IETF draft proposed standardising the header; it expired without becoming an RFC (Idempotency-Key header draft).

Store keys in a table:

CREATE TABLE idempotency_keys (    client_id      BIGINT NOT NULL,           -- keys are scoped to a client or account    idem_key       VARCHAR(255) NOT NULL,    request_hash   CHAR(64) NOT NULL,          -- fingerprint of the request body and path    status         VARCHAR(12) NOT NULL,       -- in_progress, completed    response_code  INT,    response_body  TEXT,    created_at     TIMESTAMPTZ NOT NULL DEFAULT now(),    PRIMARY KEY (client_id, idem_key));

Then, for each request:

1. Claim the key with an insert. The primary key guarantees only one request can claim it, even if two retries arrive at the same moment on different servers.

2. If the insert conflicts, the key exists:

   • Different request hash: the client reused a key for a different operation. Reject (AWS's article recommends a validation error for a parameter mismatch; 422 is a common choice).
   • Status in_progress: the original is still running. Return 409 Conflict (or wait briefly), and the client retries later.
   • Status completed: return the stored response. Do not run the operation again.
3. If the claim succeeds, perform the operation, then store the response and mark the key completed.
4. Expire keys after a retention window long enough to cover realistic retries (hours to days).

If the operation's effect is in the same database as the key table, do steps 1 to 3 in one transaction: claim the key, insert the payment record, store the response, commit. Either all happen or none do. Then a crash at any point leaves either nothing (the retry starts fresh) or everything (the retry gets the stored response).

If the effect is external (a call to a payment provider), you cannot put it in your transaction. Then:

• Record your own intent first (a payments row with status pending, keyed by your payment id).
• Call the provider with its own idempotency key, derived deterministically from your payment id, so retrying your call cannot charge twice at the provider either.
• Update your record with the provider's result. If you crash mid-way, a recovery job finds pending payments and asks the provider for their status (article 39).

This Java program models the key table in memory and shows the four outcomes: first attempt, retry, reuse with a different body, and a retry arriving while the original is still running. In a real service the map would be a database table and the steps would run in a transaction.

// File: IdempotencyDemo.javaimport java.util.HashMap;import java.util.Map;public class IdempotencyDemo {    record Stored(String requestHash, String status, String response) {}    // In production this is a database table whose primary key is the idempotency key.    static final Map<String, Stored> keys = new HashMap<>();    static int paymentsCreated = 0;    static String handle(String key, String body) {        String hash = Integer.toHexString(body.hashCode());        Stored existing = keys.get(key);        if (existing != null) {            if (!existing.requestHash().equals(hash)) {                return "422 key reused with a different request";            }            if (existing.status().equals("in_progress")) {                return "409 original request still in progress";            }            return existing.response() + " (replayed)";        }        keys.put(key, new Stored(hash, "in_progress", null));   // 1. claim the key        paymentsCreated++;                                        // 2. the side effect, done once        String response = "201 payment pay_" + paymentsCreated + " created";        keys.put(key, new Stored(hash, "done", response));      // 3. remember the result        return response;    }    public static void main(String[] args) {        String body = "{\"order\":77,\"amount\":49900}";        System.out.println("first attempt: " + handle("k-123", body));        System.out.println("retry:         " + handle("k-123", body));        System.out.println("changed body:  " + handle("k-123", "{\"order\":77,\"amount\":99900}"));        keys.put("k-456", new Stored(Integer.toHexString(body.hashCode()), "in_progress", null));        System.out.println("concurrent:    " + handle("k-456", body));        System.out.println("payments created: " + paymentsCreated);    }}

first attempt: 201 payment pay_1 createdretry:         201 payment pay_1 created (replayed)changed body:  422 key reused with a different requestconcurrent:    409 original request still in progresspayments created: 1

Four attempts, one payment. The retry receives exactly the response the first attempt produced, so the client cannot tell the difference. (A real implementation would use a cryptographic hash such as SHA-256 for the request fingerprint; Java's hashCode keeps the example short.)

• The client generates the key once per logical operation and reuses it for every retry of that operation. A new key per attempt defeats the purpose.
• Scope keys to the client or account, so two customers cannot collide.
• Do not put sensitive data in keys (Stripe's documentation makes the same point); use random values.
• Retention: long enough for the slowest realistic retry (mobile apps may retry after reconnecting hours later). Document the window.
• Validation errors that happen before any work starts need not be stored; the client fixes the request and sends it with the same or a new key.

Sometimes the data itself identifies duplicates, and a constraint does the work:

• One payment per order: UNIQUE (order_id) on successful payments.
• One like per user per photo: primary key (photo_id, user_id) (article 12).
• One booking per seat per event: UNIQUE (event_id, seat_no) (article 14).
• One welcome email per user: a sent_notifications table with UNIQUE (user_id, notification_type).

A natural key is the strongest protection because it holds regardless of which code path or client sent the duplicate. Use idempotency keys for "this request" and natural keys for "this business fact".

Queues and streams deliver at least once (article 11), so consumers must tolerate duplicates. Microsoft's Idempotent Consumer pattern describes the standard approach: record which messages have been processed, and skip repeats (Idempotent Consumer pattern).

BEGIN;INSERT INTO processed_messages (consumer, message_id) VALUES ('ledger', 'evt_7b2f9c');-- fails with a unique violation if already processed: roll back and acknowledgeUPDATE accounts SET balance = balance + 500 WHERE account_id = 9;COMMIT;-- only now acknowledge the message

The record and the effect commit together, so there is no window in which one exists without the other.

• Upserts instead of inserts (INSERT ... ON CONFLICT DO NOTHING in PostgreSQL).
• Set state instead of incrementing ("order 77 is shipped").
• Versioned updates: each event carries the entity's version; the consumer applies it only if it is newer than what it has stored. This handles duplicates and reordering at once.

UPDATE products SET price_paise = :price, version = :versionWHERE product_id = :id AND version < :version;   -- older or repeated events change nothing

An email or SMS cannot be rolled back. Options:

• Record first, then send: insert "email X for user Y" with a unique key; a sender delivers it and marks it sent. A crash between sending and marking can still send twice, so make the window small and pass a deduplication id to the provider if it supports one.
• Accept rare duplicates where the cost is low (a duplicate "your order shipped" email), but never for money.

Retries recover from transient failures. Badly designed retries turn small incidents into large ones. Microsoft's Retry pattern summarises the trade-offs (Retry pattern); the rules below are the essentials.

Retrying immediately hammers a service that is already struggling. Exponential backoff doubles the wait after each failure, with a cap. Jitter adds randomness, so thousands of clients that failed at the same moment do not all retry at the same moment.

Marc Brooker's AWS Architecture Blog post compares strategies and concludes that full jitter, sleep = random(0, min(cap, base * 2^attempt)), performs well and is simple (Exponential Backoff And Jitter). The program below prints a schedule with a 100 ms base and a 5-second cap. The random generator has a fixed seed so the output is repeatable; in production you would not fix the seed.

// File: BackoffDemo.javaimport java.util.Random;public class BackoffDemo {    public static void main(String[] args) {        long baseMs = 100;        long capMs = 5_000;        Random random = new Random(42);   // fixed seed so the output is repeatable        System.out.println("attempt  exponential   full jitter");        for (int attempt = 0; attempt < 7; attempt++) {            long exponential = Math.min(capMs, baseMs * (1L << attempt));            long jittered = (long) (random.nextDouble() * exponential); // random(0, min(cap, base * 2^attempt))            System.out.printf("%7d  %8d ms   %8d ms%n", attempt + 1, exponential, jittered);        }        int attemptsPerLayer = 3;          // 1 try + 2 retries at each layer        int layers = 3;        System.out.println(layers + " layers, " + attemptsPerLayer + " attempts each: up to "                + (int) Math.pow(attemptsPerLayer, layers) + " calls reach the bottom service");    }}

attempt  exponential   full jitter      1       100 ms         72 ms      2       200 ms        136 ms      3       400 ms        123 ms      4       800 ms        221 ms      5      1600 ms       1064 ms      6      3200 ms       2890 ms      7      5000 ms       1843 ms3 layers, 3 attempts each: up to 27 calls reach the bottom service

Notice that jittered waits are spread anywhere between zero and the exponential value, which is exactly what breaks up synchronised retry waves.

The last line of the output is the trap. If a web tier retries 3 times, the service it calls retries 3 times, and that service's database client retries 3 times, one user request can become 27 attempts at the bottom, at exactly the moment the bottom is failing. The Amazon Builders' Library article on retries discusses this amplification and recommends retrying at a single point in the stack (Timeouts, retries and backoff with jitter).

Defences:

• Retry at one layer (usually the outermost one that knows the operation is idempotent).
• Limit attempts (2 or 3 total is common for synchronous calls).
• Retry budgets: allow retries only while they are a small fraction (say 10%) of total traffic; beyond that, fail fast.
• Circuit breakers stop calling a dependency that is clearly down (article 23).
• Respect the deadline: never retry after the caller's overall deadline has passed.

Some systems offer exactly-once features inside their own boundary (for example, transactional processing within one streaming platform), but as soon as a consumer touches an external system, such as a database, an email provider or a payment provider, idempotency in your own code is what makes the result correct. In interviews, say "at-least-once delivery with idempotent consumers" rather than "exactly-once".

1. App creates idem_key = "chk_8f2c..." when checkout opens, reuses it on every retry.2. App  --POST /v1/payments  Idempotency-Key: chk_8f2c...-->  Payment API3. Payment API, in ONE database transaction:     - INSERT idempotency_keys (client, key, hash, 'in_progress')     -- claim     - INSERT payments (payment_id = pay_51, order_id = 77, status = 'pending')     COMMIT4. Payment API --charge, provider idempotency key = "pay_51"--> Provider5. Provider replies "succeeded" (or the call times out)6. Payment API, in ONE transaction:     - UPDATE payments SET status = 'succeeded' WHERE payment_id = 'pay_51'     - UPDATE idempotency_keys SET status = 'completed', response = {...}7. App <-- 201 {payment_id: pay_51, status: succeeded}Retries: - App retries step 2 after a timeout: key exists -> stored response, or 409 while in progress. - Payment API retries step 4 after a timeout: same provider key -> provider returns the original charge. - Payment API crashes between 4 and 6: a reconciliation job finds 'pending' payments older than   a minute and asks the provider for the status of "pay_51" (article 39).

Every arrow can fail and be retried, and no path charges twice.

• Retrying POST requests that create or charge without an idempotency key.
• Checking "was this processed?" and then processing, as two separate steps (a race; use a constraint).
• Returning a different response to a retry, confusing clients.
• Ignoring in_progress and processing the retry in parallel.
• Retries in every layer, with no backoff or jitter.
• Assuming "exactly-once" from a broker covers your database and emails.

• For every write API, say whether it is naturally idempotent or how you make it so.
• For payments and orders, describe the idempotency-key flow, including the provider-side key.
• For consumers, say "at-least-once with idempotent processing", and how (constraint, processed table, versions).
• When adding retries, add backoff with jitter, a limit, and "retry at one layer".

• Next: Distributed Transactions: Two-Phase Commit, the Outbox Pattern and Sagas, which makes multi-step, multi-service operations reliable using these tools.
• Later: circuit breakers and retry budgets in depth (article 23), and the full payment design (article 39).