Idempotency in API design: surviving the retry you didn't send

The support ticket says the customer was charged twice for one order. The application log says the customer checked out exactly once. Both are telling the truth, and nothing malfunctioned. The load balancer dropped a response it was allowed to drop, the HTTP client retried the way it was configured to retry, and the server did what servers do with a POST: it processed it. The bug is not in any of those components. The bug is that the API was not idempotent: it assumed a request could only arrive once, and the network never promised that.

The double charge, step by step

The sequence deserves precision, because every step in it is individually correct:

1. The client sends POST /payments for $49.
2. The server charges the card and writes the record. The operation succeeds.
3. The response is lost. A connection reset, a recycled load balancer node, a packet that lost a fight somewhere in the middle. From the client's side, "the server never got my request" and "the server succeeded and the reply evaporated" are the same observation: silence.
4. The client times out after 30 seconds and retries. This is correct behavior, and it is exactly what every retry policy you have ever enabled is for.
5. The server receives what looks like a brand new request and charges the card again. Every dashboard stays green. The customer paid $98 for a $49 order.

Without an idempotency key

client POST /payments

server charge succeeds, response lost charged $49

client silence, timeout at 30s

client retry: POST /payments

server looks brand new, charges again charged $49

$98 charged for one order

With an idempotency key

client POST /payments, Idempotency-Key: 7f3a

server charge succeeds, response stored under key charged $49

client silence, timeout at 30s

client retry: POST /payments, same key 7f3a

server key already seen, stored response returned replayed

$49 charged, retry harmless

The instinct to resist here is "we should retry less." Retries are not an edge case to be minimized; they are the default posture of every serious network client. Retry policies are one flag away in every HTTP library, and some ship enabled. Service meshes retry on your behalf without telling you. Message brokers redeliver by design, and users click the button again when the spinner takes too long. The design question is not whether the same logical request will arrive twice. It will. The question is what your API does when it does.

What HTTP method semantics already promise

HTTP thought about this decades ago. RFC 9110 defines a method as idempotent when N identical requests have the same effect on the server as one. GET, PUT, and DELETE carry that contract; POST does not. That single bit is why an off-the-shelf retry policy will happily replay a GET and hesitate over a POST. Note that the contract is about the effect, not the response: the second DELETE of a resource returns 404 where the first returned 204, but the state of the world is identical afterward, and the state is the part that matters.

So why not "just use PUT" for the payment? Because PUT is idempotent for a reason you cannot borrow without taking the rest of it. PUT means "make the resource at this URL look exactly like this body." It is a full-state replacement of something the client can name. A charge is not a document you replace. It is a side effect that ripples into systems you do not own: a card network, a ledger, a confirmation email. Plenty of operations are verbs with no natural URL to overwrite. What PUT does teach is where its safety comes from: the client names the target, so the server can tell "again" apart from "another." Keep that property and drop the replacement semantics, and you have arrived at the idempotency key.

The idempotency key pattern, done properly

The client generates a unique key per logical operation and sends it with the request:

POST /payments HTTP/1.1
Host: api.example.com
Idempotency-Key: 0b7c9d2e-4f3a-4d6b-9a1c-8e5f2a7b3c1d
Content-Type: application/json

{ "orderId": "ord_2214", "amountCents": 4900 }

The rules that make the pattern actually work:

• The key identifies the intent, not the HTTP attempt. A retry reuses the key. A genuinely new operation gets a fresh one. This distinction is the entire pattern; everything else is bookkeeping.
• The first arrival does the work and stores the outcome. The server records the key, performs the operation, then saves the response status and body against the key.
• A duplicate replays the stored response. No new work happens. The client cannot tell the replay from the original, which is precisely the point: the endpoint has become safe to retry.
• An in-flight duplicate does not run concurrently. If the retry lands while the first attempt is still executing, return 409 or hold the request until the first finishes. Running both is the double charge with extra steps.
• Keys are scoped and mortal. Scope them per operation, so a key sent to POST /payments cannot collide with one sent to POST /refunds, and expire them deliberately. Stripe keeps them for 24 hours; the exact horizon matters less than choosing one and documenting it.

Enforcement lives in the database, not a dictionary

The tempting implementation is an in-memory set of seen keys. It fails three separate ways. The second app instance behind the load balancer has its own memory and has seen nothing. A restart wipes the set at exactly the moment a deploy is causing timeouts and retries. And check-then-act races with itself: two concurrent retries both pass the "not seen yet" check, then both charge.

The mechanism that actually guarantees "at most once" is the one your database has offered since before you were born: a unique constraint.

CREATE TABLE idempotency_keys (
    operation        text        NOT NULL,
    key              uuid        NOT NULL,
    response_status  int,
    response_body    jsonb,
    created_at       timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (operation, key)
);

Claim the key before doing the work. The constraint makes exactly one caller the winner, no matter how many retries race:

var claimed = await db.TryInsertKeyAsync(operation, key);
if (!claimed)
{
    var saved = await db.GetSavedResponseAsync(operation, key);
    return saved is null
        ? Results.Conflict()   // first attempt is still running; retry later
        : Results.Json(saved.Body, statusCode: saved.Status);
}

var result = await payments.ChargeAsync(request);
await db.SaveResponseAsync(operation, key, result);
return Results.Json(result.Body, statusCode: result.Status);

Two refinements worth their cost. Where the operation writes to the same database, wrap the business write and the response save in one transaction, so no crash can record a charge without the stored response that reports it. Note what that transaction does not cover: a crash after the claim but before the work still leaves a claimed key with no outcome, and retries get 409 until the key expires. That stranded-key window is the price of the in-flight 409 behavior above, and the expiry horizon is its recovery lever. And if the work calls an external processor, pass the same key downstream; payment providers accept one for exactly this reason, so your retry does not become their duplicate.

Exactly-once is a myth, and that is fine

The uncomfortable truth underneath all of this: there is no protocol that delivers a message to a remote machine exactly once in the presence of failures. Either the sender gives up after silence (at-most-once, and you lose data) or it retries through silence (at-least-once, and you see duplicates). Every "exactly-once" feature you have read about is at-least-once delivery with deduplication bolted on at the receiver, which is the idempotency key pattern wearing a different badge.

Retries guarantee that the same request will eventually arrive twice. Idempotency is the property that makes the second arrival boring.

Message queues make this concrete. Any broker with redelivery semantics, NATS JetStream and SQS alike, promises at-least-once: if the consumer crashes after doing the work but before acknowledging, the message comes back. The consumer dedupes on the message id with, again, a unique constraint. Once your handlers are idempotent, redelivery stops being a threat and becomes the reliability mechanism itself: the system can crash anywhere and simply replay.

The retry you rehearse

You can read this pattern in ten minutes. Owning it is different. It becomes yours the first time you watch a duplicate arrive against state you are responsible for and see the unique constraint hold the line. That gap between reading and owning is where Katabench lives: the platform grades code the way production judges it, not the way a happy-path test does, and guided Labs have you build and verify against real running systems instead of diagrams.

Idempotency is also only half of the retry conversation. The other half is what the server says when it is too busy to say yes, and what a well-behaved client does with that answer, which is where rate limiting picks up. Design for the request that arrives twice. It is the only kind the network guarantees.