Skip to content
Katabench
Try free
7 min read The Katabench team

Request deadlines and retries across service boundaries

Trace a 450 ms checkout quote through three services. Set per-attempt timeouts, reserve time for the response, and decide which caller owns a retry.

A shopper changes their delivery address. The checkout page has 450 milliseconds to return a shipping quote before it offers to calculate shipping again. The quote service gives the carrier 400 milliseconds to respond. That sounds conservative until a request spends 140 milliseconds getting to the quote service. Even a successful carrier response can now arrive after the page has given up. If checkout immediately starts another request, the carrier may be calculating the same quote twice.

You can find this bug by reading a trace from the outside in. Pick the instant the answer loses its value to the caller. Work backward to the last instant each dependency can finish. Then decide where one more attempt has a chance of helping.

Start the clock where the wait begins

For this example, assume the 450 ms clock starts when the gateway accepts the quote request. It ends when the gateway hands back the response. A target for what the shopper sees on screen would also have to account for the trip from the browser and the time to render. Writing down these two endpoints stops a service-level budget from quietly becoming a promise about the whole page.

The gateway forwards the request to checkout, which looks up the basket and calls the quote service. That service calls the carrier. Reserve 50 ms across the return path for decoding the carrier response, applying the store's delivery rules, and returning the quote. That gives the carrier call a hard finish line 400 ms after the gateway took the request. The following times are an illustrative trace. Each row measures elapsed time from the same start.

Text
  0 ms   Gateway receives request. Response deadline is 450 ms.
 65 ms   Checkout has loaded the basket.
140 ms   Quote service is ready to call the carrier.
400 ms   Last acceptable finish for the carrier round trip.
450 ms   Gateway response deadline.

The carrier has at most 260 ms at dispatch. A local cap of 160 ms on the first attempt leaves room for a retry after a fast connection failure. A pool wait, DNS lookup or new connection can use some of that allowance before the carrier sees a byte. Check what the client actually times. A socket read timeout that starts after connection acquisition leaves the queue for connections unbounded. Start the deadline around the entire operation and inspect the individual timers underneath.

Leave the caller in charge of the outer clock. Within one process use an elapsed-time source that advances monotonically. Across hosts, use the protocol's deadline propagation and test that each client enables it. For example, gRPC accounts for time already spent when it forwards a deadline as a remaining timeout. Receiving applications also need to stop their own work when a call is cancelled. gRPC deadline propagation.

HTTP request

RequestAborted

Endpoint

CancellationToken

Application

pass it through

EF Core / HTTP

cancel the I/O

Miss one boundary and the signal stops there. Everything downstream keeps working.
Cancellation is a chain, not a switch at the controller. Every asynchronous boundary has to carry the same signal.

In an HTTP application that last point reaches all the way to the database client. If the endpoint accepts cancellation and the data access layer creates a fresh unrelated token, the handler may leave while its query keeps a connection occupied. A trace can show both the request deadline and the later release of that connection. Cancellation tokens in ASP.NET Core.

Price a retry at the moment of failure

Suppose measurements of successful carrier calls under the expected load led you to set a minimum useful attempt window of 90 ms. This is a policy choice. A shorter call can succeed, but this store judges the extra load a poor trade once so little time remains. Use a maximum of two total attempts and draw a small randomized backoff before considering the second. Here is one decision with a 20 ms backoff draw and the same deadlines as above.

Text
                                  Connection reset     First attempt timed out
First attempt begins                  140 ms                   140 ms
Failure arrives                       180 ms                   300 ms
Earliest second attempt               200 ms                   320 ms
Time left until carrier cutoff        200 ms                    80 ms
Second attempt (cap 160 ms)           allow                    skip

In the first column, the client can wait 20 ms and still give a second call its full 160 ms. In the second column an unconstrained retry would trample the response reserve. Compute the remaining allowance again after waiting because the scheduler can wake the task late. Cancel the wait if the caller has already left. If the carrier asks for a delay longer than the remaining budget, return the store's explicit "quote unavailable" outcome. Guessing the shipping cost would create a new checkout problem on the next screen.

If the quote service receives a call at 390 ms, it has 10 ms of carrier time left. The gateway timeout can be perfectly configured and the team still needs a rule for declining that work.

At high traffic, per-request limits alone can double the load on an unhealthy carrier. Give retries a shared allowance too. For example, let a pool of callers admit 20 additional attempts per second with a bounded burst. Their first attempts still need their own concurrency limit. Decide whether the 20 attempts are per instance or across the whole fleet. A limit of 20 in each of 30 replicas allows a sustained 600 retries per second, which is a very different conversation with the carrier. Amazon's account of overload, retries and backoff.

Pick the layer that can repeat the work

Imagine the gateway, the checkout client and the carrier client each permit two total attempts. A quick transient error at every leaf can produce eight carrier calls for one customer request. That upper bound needs no slow timeouts. Each immediate failure gives its parent time to try again. It also means a successful eighth call can make the top-level success graph look healthy while the team paying for carrier traffic sees a sharp rise in cost.

For this read-only operation the quote service owns the retry. It knows which carrier errors are transient, can repeat the exact lookup, and can return an unavailable result when the remaining time is too short. Checkout gets one answer to interpret. Give the diagram to whoever operates the gateway and ask them to point to any proxy or client library that automatically repeats a call. A retry policy in the application is only one of the places that can produce a second attempt.

Moving from a quote to buying the shipping label changes the answer. The carrier might issue the label and lose the reply. Cancelling the local call cannot revoke the label. Repeating that write requires a stable operation identifier the carrier actually honors, or a way to reconcile the unknown outcome before submitting again. An internally generated request ID that the carrier never sees cannot prevent a second purchase. Idempotency in API design.

Test the budget under contention

Use a controllable carrier stub. Have it immediately reset its first connection and return a quote on the second. Record how many attempts reached it and when the gateway responded. On the next run, delay the first attempt until the full 160 ms cap, draw the same 20 ms backoff, and verify the stub sees only one call. Repeat with 250 ms spent acquiring a checkout database connection before the quote service starts. That wait reduces the amount available to the carrier by the same amount even though the carrier itself is healthy.

Capture outcomes per logical request and per physical attempt. A rise in attempts per successful quote with flat customer latency often gives you time to investigate a degrading carrier. A rise in late completions after cancellation points to a downstream boundary that kept doing work. Keep enough trace detail to distinguish a response that arrived too late from an attempt declined locally because the budget had already run out. They create the same unavailable quote in the UI and call for different fixes.

Test the Retry Executor asks you to prove how many calls a retrying client makes and which failures stop it early. You can find it in the C# test-writing exercises. For the write-side case, System Design Studio includes Make Payment Idempotent, where a lost acknowledgement can turn a single business operation into two attempts at the provider.

Practice what you just read

More like this: C# unit testing exercises →

Get new katas and .NET tips in your inbox

A short note when fresh kata land, plus the C# and performance tricks behind the grading. No spam, unsubscribe anytime.