Saga pattern in .NET: compensating transactions, not 2PC

In the monolith, checkout was one method and one transaction. Reserve the stock, record the payment, create the order, commit. If the payment step threw, the reservation vanished with the rollback and nobody had to think about it. Then the team split inventory, payments, and orders into three services with three databases, and the first pull request wrapped three HTTP calls in a TransactionScope. It compiled. It also did nothing, because a scope in the order service has no authority over a row in the inventory database.

The incident that follows is predictable: payments captured the card, the inventory call timed out, and a customer was charged for an order that does not exist. There is no rollback to reach for. Once three writes live in three systems, undoing has to be designed as forward actions, and that design has a name.

Why two-phase commit is not the fix

The instinct is to bring the distributed transaction back. Two-phase commit is real: a coordinator asks every participant to prepare, each writes enough to guarantee it can commit, and then the coordinator says commit or abort. Its price is what rules it out between services. Between prepare and commit every participant holds its locks, and if the coordinator dies they hold them indefinitely, because a prepared participant may not decide on its own. Checkout's availability becomes the product of every participant's plus the coordinator's. And the protocol needs participants that speak it: an HTTP payment API or a card processor has no prepare phase at all. 2PC is a fine tool inside one database cluster. Across services it turns a latency problem into a locking problem.

A saga is a sequence of local transactions

A saga breaks the operation into steps, each a local transaction in one service, and pairs every step with a compensating transaction that semantically reverses it. Forward, it runs T1, T2, T3. If T3 fails, it runs C2 and then C1, in reverse, and ends in a state that is not the original but is one the business accepts.

"Semantically" is doing real work in that sentence. Compensation is not undo. Releasing a reservation restores the count, but for four seconds another customer may have been told the item was gone. Refunding a charge does not un-charge a card: the customer sees two lines on a statement, the processor keeps its fee, and settlement takes days. A compensation is a new business action that the business has agreed puts things right, so a product owner signs off on it, not only an engineer.

Microsoft's compensating transaction guidance adds the point most teams learn the hard way: a compensation can itself fail and must be retried until it succeeds. One that can be rejected for business reasons is not a compensation; it is a step that needs its own.

One checkout saga, two failures

Before the pivot you compensate; after it you only go forward

saga_id on every message

Capture fails before the pivot

compensate backward

1. T1 ok

   Reserve stock

   inventory

   →
2. T2 ok

   Authorize payment

   payments

   →
3. T3 pivot failed

   Capture payment

   payments

   →
4. T4 never ran

   Confirm order

   orders

then, in reverse order: C2 Void authorization C1 Release reservation T3 is never compensated

ends failed and consistent: nothing reserved, nothing charged

Confirm fails after the pivot

retry forward

1. T1 ok

   Reserve stock

   inventory

   →
2. T2 ok

   Authorize payment

   payments

   →
3. T3 pivot ok

   Capture payment

   payments

   →
4. T4 retrying

   Confirm order

   orders

then: re-send confirm-order with the same idempotency key, with backoff, until it succeeds

ends completed: the customer paid, so the order must exist

Orchestration or choreography

Two designs decide what runs next. With choreography, each service reacts to events: inventory hears OrderPlaced and reserves, payments hears StockReserved and authorizes, and failure events flow back the same way. There is no central component, which is attractive until the fourth service joins and nobody can say, from the code, what the full sequence is or who releases the stock when payment fails.

With orchestration, one component owns the sequence: it sends a command, waits for the reply, persists what it learned, and sends the next command, so the steps and their compensations are written in one place. The message queue vs event bus distinction maps onto this directly: orchestration is commands over queues with replies, choreography is events over a bus. Once compensation has branches or step order matters, orchestrate.

The orchestrator persists its step

The orchestrator is a state machine whose state lives in a database, not in memory. Each step is declared with a forward command and, unless it is a pivot, a compensating command:

public enum SagaState { Running, Compensating, Completed, Failed }

public sealed record SagaStep(
    string Name,
    Func<CheckoutSaga, object> Forward,
    Func<CheckoutSaga, object>? Compensate);

private static readonly SagaStep[] Steps =
[
    new("reserve-stock",
        Forward:    s => new ReserveStock(s.SagaId, s.OrderId),
        Compensate: s => new ReleaseStock(s.SagaId, s.ReservationId!.Value)),
    new("authorize-payment",
        Forward:    s => new AuthorizePayment(s.SagaId, s.CustomerId, s.Total),
        Compensate: s => new VoidAuthorization(s.SagaId, s.AuthorizationId!)),
    new("capture-payment",
        Forward:    s => new CapturePayment(s.SagaId, s.AuthorizationId!),
        Compensate: null),   // pivot: from here the saga only moves forward
    new("confirm-order",
        Forward:    s => new ConfirmOrder(s.SagaId, s.OrderId),
        Compensate: null),   // retryable, after the pivot
];

Handling a reply is a small local transaction: load the saga, check the reply is for the step you are waiting on, record what it told you, and save the saga row and the next command together. The transactional outbox is what makes "save the row and send the command" atomic; without it the orchestrator has the exact dual-write gap it was built to remove.

public async Task HandleAsync(StepSucceeded reply, CancellationToken cancellationToken)
{
    var saga = await _store.LoadAsync(reply.SagaId, cancellationToken);
    if (saga.State != SagaState.Running || Steps[saga.CurrentStep].Name != reply.Step)
    {
        return; // duplicate or stale reply
    }

    saga.Record(reply); // stores ReservationId, AuthorizationId, ...
    saga.CurrentStep++;

    if (saga.CurrentStep == Steps.Length)
    {
        saga.State = SagaState.Completed;
    }
    else
    {
        var next = Steps[saga.CurrentStep];
        _outbox.Send(next.Forward(saga), idempotencyKey: $"{saga.SagaId}:{next.Name}");
    }

    await _store.SaveAsync(saga, cancellationToken); // saga row + outbox row, one transaction
}

public async Task HandleAsync(StepFailed reply, CancellationToken cancellationToken)
{
    var saga = await _store.LoadAsync(reply.SagaId, cancellationToken);
    if (saga.State != SagaState.Running || Steps[saga.CurrentStep].Name != reply.Step)
    {
        return;
    }

    saga.State = SagaState.Compensating;
    for (var i = saga.CurrentStep - 1; i >= 0; i--) // completed steps only, in reverse
    {
        if (Steps[i].Compensate is { } undo)
        {
            _outbox.Send(undo(saga), idempotencyKey: $"{saga.SagaId}:undo:{Steps[i].Name}");
        }
    }

    await _store.SaveAsync(saga, cancellationToken);
}

The guard at the top of each handler is not paranoia. Replies arrive at least once, out of order after a retry, and occasionally for a saga that already moved on; a stale reply must be a no-op. The failure handler compensates the steps before CurrentStep, never the step that failed and never the ones that had not started.

A saga never rolls anything back. It only moves forward, and one of the directions it can move forward in happens to look like undoing.

The pivot is the point of no return

Some steps cannot be compensated at any price: a captured payment can be refunded but not voided, and a shipped parcel cannot be recalled. The first such step is the pivot transaction. Before the pivot, a failure means compensate backward. After it, a failure means retry forward until the remaining steps succeed, because the alternative is a customer who paid and has nothing. The failure handler above assumes the failed step is at or before the pivot; after it, re-send the same forward command with backoff instead of entering Compensating.

That gives you a rule for ordering steps: the ones most likely to fail go first, the irreversible ones go last. Authorize the card (voidable) before capturing it (not voidable), and reserve stock before touching money at all. The saga reference architecture uses the same vocabulary: compensable transactions, then the pivot, then retryable transactions.

Every step and every compensation is idempotent

Messages are delivered at least once, so ReserveStock will arrive twice, and so will ReleaseStock. Every command carries an idempotency key derived from the saga id and the step name, and every service records the keys it has processed in the same transaction as the work. A second AuthorizePayment with the same key returns the first authorization id instead of placing a second hold. This is the idempotency key discipline applied to internal commands; a non-idempotent compensation retried after a timeout is how a refund gets issued twice.

Timeouts, and reserve-then-expire

A saga waiting for a reply that never comes is not failed, it is stuck, and stuck sagas hold stock hostage. Two mechanisms cover it. The orchestrator schedules a timeout message when it sends a command; if the timeout arrives first, the step counts as failed and the saga compensates. And reservations expire on their own: inventory stores reserved_until and a sweeper releases anything past it, so an orchestrator that is down for an hour cannot pin stock forever. Compensation by clock is the cheapest compensation there is.

Sagas also have no isolation. Another checkout sees the reserved stock while it exists; a report sees an order in Pending. The remedies are semantic locks (a status column other readers respect) and commutative updates, worth reading next to the transaction isolation levels you gave up.

Persist the state, and put the saga id on everything

The saga table is small and boring on purpose:

CREATE TABLE checkout_sagas (
    saga_id          uuid PRIMARY KEY,
    order_id         uuid NOT NULL,
    state            text NOT NULL, -- running | compensating | completed | failed
    current_step     smallint NOT NULL DEFAULT 0,
    reservation_id   uuid,
    authorization_id text,
    version          integer NOT NULL DEFAULT 0,
    started_at       timestamptz NOT NULL DEFAULT now(),
    updated_at       timestamptz NOT NULL DEFAULT now()
);

CREATE INDEX ix_checkout_sagas_live
ON checkout_sagas (updated_at)
WHERE state IN ('running', 'compensating');

version gives optimistic concurrency between two orchestrator instances holding the same reply. The row is what you read during an incident, so its columns answer the operator's questions: which step, since when, and which ids compensate it. Put the saga id in the headers of every command, reply, and event, and in every log line their handlers write. The alert is the age of the oldest saga in running or compensating, the same shape as an outbox backlog alert, and the id lets you follow one checkout across three services without guessing.

Practice the shape before the incident does

Katabench's Architecture track has a kata for exactly this failure: "Compensate Every Completed Step", where a booking spans independently owned suppliers, one step fails partway through, and the saga must release only the steps that succeeded, in reverse order, and keep going when a release itself throws. It sits with the other architecture exercises. The System Design Studio's "Build a Reliable Checkout" series covers the same ground at the topology level with "Reserve Before Payment", "Make Payment Idempotent", "Publish Order Events Reliably", and "Operate the Reliable Checkout", each graded structurally against authored rules for required paths and forbidden bypasses. The Studio canvas is where the order of steps becomes visible, which makes it the cheapest place to discover you put the pivot first.