Request deadlines and retries across service boundaries
Trace a 450 ms checkout quote through three services. Set per-attempt timeouts, reserve time for the response, and decide which caller owns a retry.
A shopper changes their delivery address. The checkout page has 450 milliseconds to return a shipping quote before it offers to calculate shipping again. The quote service gives the carrier 400 milliseconds to respond. That sounds conservative until a request spends 140 milliseconds getting to the quote service. Even a successful carrier response can now arrive after the page has given up. If checkout immediately starts another request, the carrier may be calculating the same quote twice.
You can find this bug by reading a trace from the outside in. Pick the instant the answer loses its value to the caller. Work backward to the last instant each dependency can finish. Then decide where one more attempt has a chance of helping.
For this example, assume the 450 ms clock starts when the gateway accepts the quote request. It ends when the gateway hands back the response. A target for what the shopper sees on screen would also have to account for the trip from the browser and the time to render. Writing down these two endpoints stops a service-level budget from quietly becoming a promise about the whole page.
The gateway forwards the request to checkout, which looks up the basket and calls the quote service. That service calls the carrier. Reserve 50 ms across the return path for decoding the carrier response, applying the store's delivery rules, and returning the quote. That gives the carrier call a hard finish line 400 ms after the gateway took the request. The following times are an illustrative trace. Each row measures elapsed time from the same start.
0 ms Gateway receives request. Response deadline is 450 ms.
65 ms Checkout has loaded the basket.
140 ms Quote service is ready to call the carrier.
400 ms Last acceptable finish for the carrier round trip.
450 ms Gateway response deadline.The carrier has at most 260 ms at dispatch. A local cap of 160 ms on the first attempt leaves room for a retry after a fast connection failure. A pool wait, DNS lookup or new connection can use some of that allowance before the carrier sees a byte. Check what the client actually times. A socket read timeout that starts after connection acquisition leaves the queue for connections unbounded. Start the deadline around the entire operation and inspect the individual timers underneath.
Leave the caller in charge of the outer clock. Within one process use an elapsed-time source that advances monotonically. Across hosts, use the protocol's deadline propagation and test that each client enables it. For example, gRPC accounts for time already spent when it forwards a deadline as a remaining timeout. Receiving applications also need to stop their own work when a call is cancelled. gRPC deadline propagation.
HTTP request
RequestAborted
Endpoint
CancellationToken
Application
pass it through
EF Core / HTTP
cancel the I/O
In an HTTP application that last point reaches all the way to the database client. If the endpoint accepts cancellation and the data access layer creates a fresh unrelated token, the handler may leave while its query keeps a connection occupied. A trace can show both the request deadline and the later release of that connection. Cancellation tokens in ASP.NET Core.
Suppose measurements of successful carrier calls under the expected load led you to set a minimum useful attempt window of 90 ms. This is a policy choice. A shorter call can succeed, but this store judges the extra load a poor trade once so little time remains. Use a maximum of two total attempts and draw a small randomized backoff before considering the second. Here is one decision with a 20 ms backoff draw and the same deadlines as above.
Connection reset First attempt timed out
First attempt begins 140 ms 140 ms
Failure arrives 180 ms 300 ms
Earliest second attempt 200 ms 320 ms
Time left until carrier cutoff 200 ms 80 ms
Second attempt (cap 160 ms) allow skipIn the first column, the client can wait 20 ms and still give a second call its full 160 ms. In the second column an unconstrained retry would trample the response reserve. Compute the remaining allowance again after waiting because the scheduler can wake the task late. Cancel the wait if the caller has already left. If the carrier asks for a delay longer than the remaining budget, return the store's explicit "quote unavailable" outcome. Guessing the shipping cost would create a new checkout problem on the next screen.
If the quote service receives a call at 390 ms, it has 10 ms of carrier time left. The gateway timeout can be perfectly configured and the team still needs a rule for declining that work.
At high traffic, per-request limits alone can double the load on an unhealthy carrier. Give retries a shared allowance too. For example, let a pool of callers admit 20 additional attempts per second with a bounded burst. Their first attempts still need their own concurrency limit. Decide whether the 20 attempts are per instance or across the whole fleet. A limit of 20 in each of 30 replicas allows a sustained 600 retries per second, which is a very different conversation with the carrier. Amazon's account of overload, retries and backoff.
Imagine the gateway, the checkout client and the carrier client each permit two total attempts. A quick transient error at every leaf can produce eight carrier calls for one customer request. That upper bound needs no slow timeouts. Each immediate failure gives its parent time to try again. It also means a successful eighth call can make the top-level success graph look healthy while the team paying for carrier traffic sees a sharp rise in cost.
For this read-only operation the quote service owns the retry. It knows which carrier errors are transient, can repeat the exact lookup, and can return an unavailable result when the remaining time is too short. Checkout gets one answer to interpret. Give the diagram to whoever operates the gateway and ask them to point to any proxy or client library that automatically repeats a call. A retry policy in the application is only one of the places that can produce a second attempt.
Moving from a quote to buying the shipping label changes the answer. The carrier might issue the label and lose the reply. Cancelling the local call cannot revoke the label. Repeating that write requires a stable operation identifier the carrier actually honors, or a way to reconcile the unknown outcome before submitting again. An internally generated request ID that the carrier never sees cannot prevent a second purchase. Idempotency in API design.
Use a controllable carrier stub. Have it immediately reset its first connection and return a quote on the second. Record how many attempts reached it and when the gateway responded. On the next run, delay the first attempt until the full 160 ms cap, draw the same 20 ms backoff, and verify the stub sees only one call. Repeat with 250 ms spent acquiring a checkout database connection before the quote service starts. That wait reduces the amount available to the carrier by the same amount even though the carrier itself is healthy.
Capture outcomes per logical request and per physical attempt. A rise in attempts per successful quote with flat customer latency often gives you time to investigate a degrading carrier. A rise in late completions after cancellation points to a downstream boundary that kept doing work. Keep enough trace detail to distinguish a response that arrived too late from an attempt declined locally because the budget had already run out. They create the same unavailable quote in the UI and call for different fixes.
Test the Retry Executor asks you to prove how many calls a retrying client makes and which failures stop it early. You can find it in the C# test-writing exercises. For the write-side case, System Design Studio includes Make Payment Idempotent, where a lost acknowledgement can turn a single business operation into two attempts at the provider.
Practice what you just read
- Open in the editor: Test the Retry Executor
Test writing Hard Free, no account needed
Test the Retry Executor
You're given a correct retry executor. Write tests that prove its attempt budget, exception policy, and call sequence.
- Open the Pro kata: Make Payment Idempotent
System Design Studio Medium Katabench Pro
Make Payment Idempotent
Queue redelivery is charging customers twice. Give every payment worker the same durable idempotency ledger and provider path.
More like this: C# unit testing exercises →