Distributed Transactions: Two-Phase Commit, the Outbox Pattern and Sagas

Placing an order looks like one action to a customer. Inside an e-commerce system it is several: create the order, reserve stock, charge the card, schedule shipping, and tell other parts of the system that it happened. In a single database, a transaction would make all of that atomic. But the order lives in the order service's database, stock in the inventory service, payments with a provider, and events in a message broker. There is no single transaction that spans them.

What happens when the card is charged and then the shipping step fails? Or the order is saved and the "order placed" event is never published? This article covers the three main tools for these situations: the transactional outbox for "database plus event", two-phase commit for true cross-system atomicity (and why it is rarely used between services), and sagas for long business processes across services.

• Why local transactions stop at the edge of one database.
• The dual-write problem, step by step.
• The transactional outbox pattern, relays, change data capture and the inbox on the consumer side.
• Two-phase commit: how it works and why it blocks.
• Sagas: compensating transactions, choreography versus orchestration, pivot and retryable steps.
• The anomalies sagas allow and the countermeasures.
• How to choose between these approaches, with a runnable saga example.

A database transaction (article 14) is atomic because one system controls everything it touches: its own log, its own locks, its own commit. The moment an operation involves two systems that commit independently, such as your database and a message broker, or your database and another service's database, no single commit covers both.

This shows up in two shapes:

1. Database plus event: save data and publish a message about it.
2. Several services: a business process that changes data owned by several services.

Pixly's upload API (article 05) saves a photo row and publishes photo.uploaded. Written naively:

1. INSERT photo row into the database   (commit)2. Publish "photo.uploaded" to the broker

What if the process crashes between 1 and 2? The photo exists, but no event is ever published: no thumbnails, no feed update, forever. Swap the order:

1. Publish "photo.uploaded"2. INSERT photo row  (crash or failure here)

Now consumers receive an event about a photo that does not exist. Wrapping both in a database transaction does not help, because the broker is not part of it: the publish happens immediately, while the database commit can still fail.

Every variation has a gap. This is the dual-write problem: writing to two systems without a shared transaction means a crash can leave them disagreeing. It is subtle because it only fails occasionally, under crashes or timeouts, which is exactly when nobody is watching.

The fix is to write to one system atomically, and move the second write out of the request:

1. In one local transaction, write the business change and an outbox row describing the event, in the same database.
2. A separate relay reads unsent outbox rows, publishes them to the broker, and marks them sent.

The microservices.io description of the Transactional Outbox pattern presents exactly this: the service inserts messages into an outbox table as part of the local transaction, and a message relay publishes them (Transactional outbox). Microsoft's guidance describes the same pattern for Azure services (Transactional Outbox pattern with Azure Cosmos DB).

CREATE TABLE outbox (    event_id      UUID PRIMARY KEY,    aggregate_id  BIGINT NOT NULL,          -- for example the photo id; used for ordering    event_type    VARCHAR(100) NOT NULL,    payload       JSONB NOT NULL,    created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),    published_at  TIMESTAMPTZ);BEGIN;INSERT INTO photos (photo_id, owner_id, status, ...) VALUES (9001, 42, 'processing', ...);INSERT INTO outbox (event_id, aggregate_id, event_type, payload)VALUES ('7b2f9c...', 9001, 'photo.uploaded', '{"photo_id": 9001, "owner_id": 42}');COMMIT;

REQUEST PATH                         ASYNC PATHAPI --> one transaction:             Relay --reads unsent outbox rows--> Broker --> consumers        photos row + outbox row             --marks them published-->

Either both rows commit or neither does. If the relay crashes after publishing but before marking a row sent, it publishes the row again on restart. So outbox publication is at-least-once, and consumers must be idempotent (article 20), using event_id to deduplicate.

• Polling relay: a job queries WHERE published_at IS NULL ORDER BY created_at every short interval, publishes, and updates. Simple; adds some database load and latency.
• Change data capture (CDC): a connector reads the database's own change log (in PostgreSQL, the write-ahead log) and turns inserted outbox rows into messages. Lower latency and no polling queries; another component to operate. Tools such as Debezium are commonly used for this.

• To keep events for one entity in order, publish them in created_at order per aggregate_id, and use the aggregate id as the message key (so a partitioned broker keeps them in one partition, article 11).
• Delete or archive published outbox rows periodically so the table stays small.

The receiving service often mirrors the pattern: record the incoming event_id in an inbox (processed messages) table in the same transaction as the effect, and skip duplicates. This is the idempotent consumer from article 20 under another name; Microsoft describes it as the Idempotent Consumer pattern (Idempotent Consumer pattern). Outbox on the sender plus inbox on the receiver gives effectively-once processing across services.

What if you need several independent databases to commit atomically: all or none, even across machines? The classic protocol is two-phase commit, run by a coordinator:

Phase 1 (prepare / vote):  Coordinator --"prepare"--> Participant A: writes changes durably, holds locks, votes YES  Coordinator --"prepare"--> Participant B: votes YES   (any NO -> abort everyone)Phase 2 (commit):  Coordinator records "commit" durably, then  Coordinator --"commit"--> A and B: make changes permanent, release locks

Once a participant votes yes, it has promised to commit if told to, so it must keep its locks and wait for the coordinator's decision.

The blocking problem. If the coordinator crashes after the participants voted yes but before they learn the decision, they are stuck: they cannot commit (maybe the decision was abort) and cannot abort (maybe it was commit). They hold their locks until the coordinator recovers. Other transactions touching those rows wait too.

Costs of 2PC:

• Latency: at least two round trips, plus durable writes, for every transaction.
• Availability: the transaction needs every participant and the coordinator; it is only as available as all of them together (article 02's availability in series).
• Coupling: every participant must support the protocol (for example the XA standard) and trust one coordinator.
• Lock holding during the protocol reduces throughput under contention.

Where 2PC is used: inside distributed databases (which pair it with consensus-replicated participants so a coordinator failure does not block indefinitely), and occasionally between a few systems within one organisation at modest volume. Between independently deployed microservices, owned by different teams and using different stores, it is rarely a good fit, which is why sagas exist.

A saga breaks a business transaction into a sequence of local transactions, one per service. Each step commits on its own and triggers the next. If a step fails, the saga runs compensating transactions to undo the effects of the steps that already completed. Microsoft's Saga pattern guidance describes exactly this structure, along with its two coordination styles and the risks it introduces (Saga design pattern), as does the microservices.io description (Saga).

Order saga:  1. Order service:     create order (PENDING)          compensate: mark CANCELLED  2. Inventory service: reserve stock                   compensate: release stock  3. Payment service:   charge card                     compensate: refund  4. Order service:     mark CONFIRMED  5. Shipping service:  schedule delivery               (retry until it succeeds)

A compensating transaction is a new action that semantically undoes an earlier one, and it is visible: a refund appears on the customer's statement next to the charge; a cancelled order exists in history. Some actions cannot be compensated at all (an email cannot be unsent, goods cannot be un-shipped). This leads to an important design rule from the Saga pattern guidance:

• Compensable steps come first: they can be undone.
• The pivot step is the point of no return. Once it succeeds, the saga must complete.
• Retryable steps come after the pivot. They must eventually succeed, so they are idempotent and retried until done.

Order the steps so that irreversible actions come last. Charge the card (refundable) before sending "your order is confirmed"; schedule shipping only after payment succeeds.

Choreography: each service listens for events and decides what to do next. No central controller.

Order svc --OrderCreated--> Inventory svc --StockReserved--> Payment svc --PaymentCaptured--> Order svc                                     \--StockFailed--> Order svc (cancel)

Orchestration: an orchestrator (a service or workflow engine) holds the saga's state and tells each participant what to do.

                 Orchestrator (saga state stored durably)       /---- reserve ---->  Inventory      /<---- reserved -----     /------ charge ----->  Payment    /<------ failed ------   \------- release ----->  Inventory   (compensation)

These benefits and drawbacks follow Microsoft's comparison in the Saga pattern guidance. A practical rule: choreography for two or three steps; orchestration beyond that, often using a durable workflow engine (products such as Temporal, AWS Step Functions or Azure Durable Functions are examples) so that saga state survives crashes and timeouts are handled for you.

This Java program models an orchestrator: it runs steps in order, and when one fails, runs the compensations of the completed steps in reverse order. Each step stands for a local transaction in a different service. The first run declines the card; the second succeeds.

// File: SagaDemo.javaimport java.util.ArrayDeque;import java.util.Deque;import java.util.List;public class SagaDemo {    // One saga step: a local transaction in some service, and the action that undoes its effect.    record Step(String name, Runnable action, Runnable compensation) {}    static boolean cardDeclined = true;    static void runSaga(List<Step> steps) {        Deque<Step> completed = new ArrayDeque<>();        for (Step step : steps) {            try {                step.action().run();                completed.push(step);            } catch (RuntimeException e) {                System.out.println("  FAILED: " + step.name() + " (" + e.getMessage() + ")");                while (!completed.isEmpty()) {                    completed.pop().compensation().run();   // undo completed steps, newest first                }                System.out.println("  saga finished: order cancelled");                return;            }        }        System.out.println("  saga finished: order confirmed");    }    static void chargeCard() {        if (cardDeclined) {            throw new IllegalStateException("card declined");        }        System.out.println("  payment captured");    }    public static void main(String[] args) {        List<Step> steps = List.of(                new Step("create order",                        () -> System.out.println("  order 77 created as PENDING"),                        () -> System.out.println("  compensate: order 77 marked CANCELLED")),                new Step("reserve stock",                        () -> System.out.println("  2 units reserved"),                        () -> System.out.println("  compensate: 2 units released")),                new Step("charge payment",                        SagaDemo::chargeCard,                        () -> System.out.println("  compensate: payment refunded")),                new Step("confirm order",                        () -> System.out.println("  order 77 CONFIRMED"),                        () -> { }));        System.out.println("Run 1, card declined:");        runSaga(steps);        cardDeclined = false;        System.out.println("Run 2, card accepted:");        runSaga(steps);    }}

Run 1, card declined:  order 77 created as PENDING  2 units reserved  FAILED: charge payment (card declined)  compensate: 2 units released  compensate: order 77 marked CANCELLED  saga finished: order cancelledRun 2, card accepted:  order 77 created as PENDING  2 units reserved  payment captured  order 77 CONFIRMED  saga finished: order confirmed

A production orchestrator adds what this model leaves out: it persists saga state after every step (so a crash resumes instead of restarting), it retries steps and compensations with backoff, every step and compensation is idempotent (because retries repeat them), and steps have timeouts so a silent participant does not hang the saga forever.

A refund call can fail too. Compensations must be retried until they succeed (they are retryable steps), with alerts if they keep failing, and a manual process as the last resort. This is one reason to keep sagas short and compensations simple.

Because each step commits independently, other transactions can see a saga's intermediate states. Microsoft's guidance lists the typical anomalies: lost updates (one saga overwrites another's changes), dirty reads (reading changes another saga will later compensate) and fuzzy or non-repeatable reads (steps of one saga reading inconsistent data). It also lists countermeasures (Saga design pattern):

In practice, the most important habit is the semantic lock: model intermediate states (PENDING, RESERVED, PAYMENT_PENDING) explicitly, and make every other process respect them.

Services: Order, Inventory, Payment (calls an external provider), Shipping, Notification.

Design:

1. POST /v1/orders (with an idempotency key) runs one local transaction in the Order service: insert the order as PENDING and an outbox row OrderCreated. Respond 202 Accepted with the order id; the client polls or receives a push when the status changes.
2. An orchestrator (or the Order service acting as one) consumes OrderCreated and drives the saga, storing its state in a checkout_sagas table.
3. Reserve stock: Inventory reserves with a conditional update, recording the reservation keyed by order id (idempotent). On "insufficient stock", the saga cancels the order.
4. Charge: Payment creates a payment record and calls the provider with a provider idempotency key derived from the order (article 20). This is the pivot. If declined: release stock, cancel order.
5. After the pivot: confirm the order, schedule shipping, notify the customer. These are retryable; they are retried until they succeed. Notification is last because an email cannot be taken back.
6. Timeouts: if payment stays pending past a deadline (for example the provider is slow), the orchestrator queries the provider for the final status rather than guessing.

Consistency the user sees: the order shows PENDING for a moment, then CONFIRMED or CANCELLED. Stock may appear reserved for a few seconds before a declined payment releases it. Both are acceptable, and both are visible states in the UI rather than hidden glitches.

• Publishing events right after a database commit in application code and calling it reliable.
• Treating compensation as a database rollback, and putting irreversible steps first.
• Non-idempotent saga steps or compensations, so retries double effects.
• No timeouts on saga steps.
• Hiding intermediate states from users and from other processes.
• Splitting services so finely that every operation needs a saga.

• When you write to a database and publish an event, say "transactional outbox", and that consumers are idempotent.
• When a process spans services, propose a saga, list the steps with compensations, and point out the pivot.
• Say whether it is orchestrated or choreographed, and why.
• If asked about 2PC, explain the blocking problem and why you avoid it across services.

@Transactionalvoid placeOrder(order) {    orderRepository.save(order);    broker.publish("OrderPlaced", order);    // inside the database transaction    inventoryClient.reserve(order.items);    // HTTP call to another service}

CREATE TABLE outbox (    event_id     UUID PRIMARY KEY,    account_id   BIGINT NOT NULL,          -- ordering key    event_type   VARCHAR(50) NOT NULL,    payload      JSONB NOT NULL,    created_at   TIMESTAMPTZ NOT NULL DEFAULT now(),    published_at TIMESTAMPTZ);CREATE INDEX outbox_unpublished ON outbox (created_at) WHERE published_at IS NULL;

• Next: Distributed Locks, Leases, Leader Election and Consensus, for the cases where only one node may act at a time.
• Later: event-driven architecture builds on the outbox (article 27); the payment (39) and food-delivery (42) case studies use sagas end to end.