Service level objectives for asynchronous work
Build an upload SLO around the wait users experience. Count jobs that never complete, set the deadline, and calculate how fast failures spend the error budget.
A customer uploads the packing list for a truck that is about to leave. The API acknowledges the file in 90 milliseconds. Ten minutes later the page still says "checking file." The API dashboard reports a successful request. The scanner dashboard reports that every scan it finished today took less than a second. Both measurements can be accurate while the customer is stuck.
The upload has two promises with different clocks. The first covers getting the bytes into the service and acknowledging receipt. The second covers how long the owner must wait to find out whether the file is ready to share. A service level indicator (SLI) measures one of those promises. A service level objective (SLO) gives it a target over a window. You need the distinction most when the API has already finished its work and a queue has quietly stopped delivering messages. Google's SRE workbook on implementing SLOs.
Follow the path marked "Direct to storage" in the diagram. The client sends the bytes to storage and asks the API to finalize the upload. The API verifies the object, acknowledges it and arranges for scanning. A decision eventually reaches the status endpoint the owner is polling. A missing message between finalization and scanning can leave the file there indefinitely.
Consider the following objective for a service that mostly handles small dispatch documents. The numbers are design assumptions. A team choosing them would check upload sizes and how long the warehouse can wait before issuing paperwork another way.
Eligible event First successful finalization of an upload up to 20 MB
Good event A scan verdict is visible to the owner within 120 seconds
Valid verdicts Available, or quarantined with a policy reason
Window and goal At least 99.5% good over a rolling 28 days-
Dispatch A
- 10:00:00 accepted
- 10:01:59 available
The verdict became visible after 119 seconds. Count one good upload.
-
Dispatch B
- 10:00:00 accepted
- 10:02:01 available
The verdict arrived a second late. The completed scan still misses the objective.
-
Dispatch C
- 10:00:00 accepted
- No verdict
At 10:02:01 the deadline has passed. Count this upload even if the worker never saw it.
Start with the committed acceptance record for a distinct upload ID. An HTTP retry of finalization belongs to that same upload. An abandoned file whose owner never finalized it falls outside this processing objective. If the customer cannot get the server to accept the finalization in the first place, a separate acceptance indicator has to capture that failure. Keeping both indicators on the same dashboard helps avoid a particularly misleading success story when acceptance volume collapses.
Quarantine with a genuine scanner verdict is a timely decision the owner can act on. A scanner crash that becomes a generic "failed" status is a bad result. Record policy decisions separately from technical failures, then look at the mix of outcomes. A release that quarantines every clean file could satisfy a timeliness metric and make the service useless. A known-clean document through the full upload and download path gives the team a direct check on that class of regression.
Imagine grading today's work from the scanner's completion log. A lost message produces no log entry. A file that waits overnight appears tomorrow and makes today's numbers look better. The denominator for this indicator lives upstream of both failures. Start with every eligible upload whose two-minute deadline has passed, and look up the first qualifying verdict for each one.
Suppose accepted_uploads has one row per upload and records server-side acceptance time and
verified size in bytes. Here, 20 MB means 20,000,000 bytes. The table owner_verdicts is the
read model behind the authenticated status endpoint. Its visible_at records when the verdict
became readable there. A worker that logs "scan done" before this update has finished only part
of the journey. The following PostgreSQL query scores a complete cohort between two acceptance
times. Both bounds must be at least two minutes old when the query runs.
WITH eligible AS (
SELECT id, accepted_at
FROM accepted_uploads
WHERE accepted_at >= :window_start
AND accepted_at < :window_end
AND size_bytes <= 20000000
)
SELECT count(*) AS eligible,
count(*) FILTER (WHERE EXISTS (
SELECT 1
FROM owner_verdicts v
WHERE v.upload_id = e.id
AND (v.status = 'available' OR
(v.status = 'quarantined' AND v.policy_reason IS NOT NULL))
AND v.visible_at >= e.accepted_at
AND v.visible_at <= e.accepted_at + interval '120 seconds'
)) AS good
FROM eligible e;A row with no verdict stays in the denominator and contributes zero good events. Several worker deliveries for the same upload contribute at most one. A verdict exactly at the 120-second boundary passes. Use timestamp fields with the same clock basis and retain verdict history if a subsequent state transition can overwrite the first visible decision. For the rolling report, score the 28 days ending two minutes ago so even the most recent row had its full chance to pass.
This measurement has a blind spot. The status read model may contain a verdict that the owner cannot retrieve because the status endpoint is failing. Check the public route with an authorized test account and track read errors independently. If tests reveal material delay between that read model and what callers see, instrument a closer observation point and evaluate the two measures side by side. Silently switching the finish timestamp halfway through the reporting window makes a metric change look like a service improvement.
A lost scan job still represents a customer waiting. Its upload ID belongs in the denominator even when every worker log is empty.
Suppose one completed 28-day window contains 40,000 eligible uploads. At a 99.5% objective it can contain at most 200 bad uploads. That's the error budget for that population. It is denominated in uploads, so translating it into minutes of allowed downtime would require an assumption about when failures happen and how many documents arrive then.
Take a smaller incident cohort within that same window. Of 2,000 eligible uploads, 1,960 got a verdict within two minutes, 24 got one late, and 16 never got one at all. The bad fraction was 40 divided by 2,000, or 2%. The allowed bad fraction is 0.5%, giving a burn rate of four for this incident. Those 40 files used 20% of the window's 200-file allowance. They still count if the delayed worker recovers tonight and every file eventually becomes available.
A burn rate also needs traffic context. Four times the allowed rate at shift change can affect hundreds of people. The same rate on a quiet weekend may be based on a handful of documents. For a sustained problem, compare a longer window that establishes material impact with a short window that tells the operator whether customers are still affected. For a low-volume route, a synthetic upload exercises it between real jobs. Keep synthetic traffic out of the customer denominator so a healthy probe cannot dilute a real customer's failure. Google's SRE workbook on error budget alerts.
Look up one of the 24 late upload IDs and one of the 16 missing ones. If the late files spent most of their time in a queue before any scan started, speeding up the status endpoint will barely move the customer metric. For a missing file, trace whether the acceptance row had a corresponding durable message. More workers cannot process an event that the system never published. Keep publish lag, oldest waiting job and worker saturation available to explain the customer metric. Designing a file storage service.
Check a regional breakdown too. One warehouse cut off from its queue can disappear in an otherwise healthy global ratio. Calculate the global result by summing good and eligible counts across sites. A simple average of percentages gives a site with ten uploads the same weight as one with ten thousand. Keep the site-level view alongside the global objective when a single stuck dock is enough to need an operator.
Katabench's Operate the Upload Pipeline exercise in System Design Studio asks where telemetry belongs along the asynchronous path. To practice the denominator itself, The Outage at 0% Errors in the C# coding challenges gives you a metrics job whose inputs contain the failed requests its dashboard lost.
Practice what you just read
- Open in the editor: The Outage at 0% Errors
Algorithm Medium Free, no account needed
The Outage at 0% Errors
A route failed for 40 minutes while the dashboard showed 0% errors, so the alert never fired. Where did 412 failed requests go?
- Open the Pro kata: Operate the Upload Pipeline
System Design Studio Hard Katabench Pro
Operate the Upload Pipeline
Turn a fragile file upload into a secure, observable asynchronous pipeline built for horizontal scale.
More like this: C# coding challenges →