Debugging production issues: a method for the first hour
Debugging production issues starts with numbers. A simulated webhook outage traces to one tenant: 4.1% of attempts holding 91% of worker time.
The ticket says "webhooks are late." Someone in the channel mentions the deploy that went out this afternoon, somebody else opens the delivery worker's source, and a third person has started a rollback. Twenty minutes later three people hold three theories and nobody can say how late is late.
Debugging production issues goes badly in a predictable way: the first plausible story gets all the effort. The cure is a fixed order of work that makes a story earn its place. Each step below produces something the next step needs.
The scenario is invented and the numbers are real output: a small discrete-event simulation of a webhook service, queried in PostgreSQL. Ten workers pull from one first-in, first-out queue at about 25 deliveries per second, across 40 tenants.
A symptom you can act on names a quantity, a threshold, a start time and a scope. Start with the two things a delivery does, waiting and attempting, and look at more than the middle of each:
SELECT to_char(date_bin('5 minutes', enqueued_at, '2026-11-01'), 'HH24:MI') AS enqueued,
count(*) AS deliveries,
round(percentile_cont(0.95) WITHIN GROUP (ORDER BY extract(epoch FROM
coalesce(started_at, '2026-11-01 14:30') - enqueued_at))::numeric, 1) AS p95_wait_s,
round(percentile_cont(0.5) WITHIN GROUP (ORDER BY extract(epoch FROM
finished_at - started_at))::numeric, 2) AS p50_attempt_s,
round(percentile_cont(0.99) WITHIN GROUP (ORDER BY extract(epoch FROM
finished_at - started_at))::numeric, 2) AS p99_attempt_s
FROM deliveries GROUP BY 1 ORDER BY 1; enqueued | deliveries | p95_wait_s | p50_attempt_s | p99_attempt_s
----------+------------+------------+---------------+---------------
14:00 | 7582 | 0.0 | 0.12 | 0.26
14:05 | 7552 | 0.0 | 0.12 | 0.28
14:10 | 7591 | 684.9 | 0.12 | 30.00
14:15 | 7497 | 843.4 | 0.12 | 30.00
14:20 | 7433 | 584.4 | |
14:25 | 7625 | 286.1 | |"Webhooks are late" has become a claim you can test: p95 queue wait went from zero to over ten minutes while arrivals did not change. The p99 column guards against a near miss. Read alone, the flat median says attempts are fine and sends you straight to the queue, but a flat median rules out only a broad slowdown, because a small tenant cannot move it. At the 99th percentile, attempts jumped from under 0.3 seconds to exactly 30 seconds in the same bucket as the wait. Two things changed at once, and any explanation has to cover both.
The last two rows look like recovery, and they are an artifact. At 14:30, 21,307 deliveries were still
queued. A row enqueued at 14:25 can only have waited five minutes so far, and without the coalesce it
would be missing from the percentile altogether. Those rows have no attempt columns for the same reason.
Any metric computed only from finished work flatters an outage while it is happening, so ask what your
dashboard does with the requests still in flight.
The boundary is your strongest clue, because something changed there. Google's SRE book makes the same point about recent changes: a system tends to keep behaving as it was until something pushes it, whether a config change or a shift in load. Rerun the query per minute around the edge:
enqueued | deliveries | p95_wait_s
----------+------------+------------
14:08 | 1468 | 0.00
14:09 | 1488 | 0.00
14:10 | 1565 | 124.61
14:11 | 1490 | 311.24The first delivery to wait more than a second was enqueued at 14:10:07. The rolling deploy of release v84 started at 14:05:00, one worker every 90 seconds. That gap of five minutes does not clear the deploy, but it does make you want a better reason than "it was recent."
Now look for other events in the same window, and write down what you find along with the time you looked. In this data, the first delivery attempt to hit the 30-second timeout started at 14:10:02.
A hypothesis earns attention by naming what you would see if it were false. "The deploy did it" names nothing. "Attempts on v84 workers are slower or fail more often than attempts on v83 workers in the same minutes" can lose. Writing the prediction first also protects you from reading a result generously after the fact. The SRE book lists the related traps, among them latching on to the causes of past problems and chasing correlations that are coincidence.
Rank candidates by how cheaply you can test them, not by how much you like them, since a quick elimination is worth more than a slow confirmation. A clear negative result counts as progress, which the same chapter argues at length.
Write the prediction before you run the query
Release v84 broke delivery
If true
Attempts on v84 workers are slower or fail more than attempts on v83 workers in the same minutes
Measured
Same 0.12 s median and same 4.5% timeouts
Ruled out
Customer endpoints are slow everywhere
If true
Attempt duration rises across tenants
Measured
39 tenants unchanged at a 0.12 s median
Ruled out
One tenant is holding the workers
If true
One tenant's share of slot time far exceeds its share of attempts
Measured
4.1% of attempts, 91% of slot time
Survives
Source code tells you what could happen. Data tells you what did, and it narrows which code deserves your reading. Comparing the two releases over the minutes when both were running keeps time out of the comparison, an easy mistake here, because a naive split would compare v83's early, lightly queued attempts with v84's later ones:
SELECT release, count(*) AS attempts,
round(percentile_cont(0.5) WITHIN GROUP (ORDER BY extract(epoch FROM
finished_at - started_at))::numeric, 2) AS p50_attempt_s,
round(100.0 * count(*) FILTER (WHERE outcome = 'timeout') / count(*), 1) AS timeout_pct
FROM deliveries
WHERE started_at >= '2026-11-01 14:12' AND started_at < '2026-11-01 14:16'
GROUP BY release ORDER BY release; release | attempts | p50_attempt_s | timeout_pct
---------+----------+---------------+-------------
v83 | 575 | 0.12 | 4.5
v84 | 992 | 0.12 | 4.5The release hypothesis is out, and the rollback in progress would have cost time for nothing. Splitting attempt time by tenant settles the second hypothesis and suggests the third:
who | attempts | p50_attempt_s | p95_attempt_s
-----------+----------+---------------+---------------
other 39 | 7640 | 0.12 | 0.22
tenant-17 | 327 | 30.00 | 30.00Brendan Gregg's USE method supplies the next question: for each resource, check utilization, saturation and errors. Worker slots are the resource here, and the queue is their saturation. Summing attempt duration per tenant since 14:12 shows who holds them:
tenant | attempts | pct_of_attempts | slot_seconds | pct_of_slot_time
-----------+----------+-----------------+--------------+------------------
tenant-17 | 327 | 4.1 | 9810 | 91.0
tenant-07 | 226 | 2.8 | 29 | 0.3tenant-17 made 4.1% of the attempts and held 91% of the worker time, while the median attempt stayed at 0.12 seconds.
The arithmetic agrees with the query. tenant-17 sends about one delivery per second, and each one now holds a worker for 30 seconds, so it demands 30 busy workers from a pool of 10. Every other tenant's deliveries wait behind those hanging attempts in one shared line, which is why a failure at one customer's endpoint looks like an outage for all 40. It also explains the p99 jump: at about 4% of attempts, this tenant fills everything above the 96th percentile and never reaches the median.
A story that fits the past can still be wrong, so make it predict a result you have not seen. If slot starvation is the cause, then capping each tenant at two concurrent attempts should bring other tenants back to their old wait times while tenant-17's own queue grows. The same simulation with that cap:
| At 14:30 | One shared queue | Two slots per tenant |
|---|---|---|
| Other tenants' deliveries still queued | 20,424 | 0 |
| tenant-17 deliveries still queued | 883 | 1,170 |
| Other tenants' longest wait since 14:10 | 851.5 s | 0.172 s |
The prediction held, including the part that costs something: tenant-17's backlog grew. In a live incident this step is your mitigation as well, and you should apply the smallest reversible version first. A circuit breaker and a shorter timeout are the natural follow-ups, and queue-based load leveling explains why a shared queue moves pressure around without removing it.
The method depends on notes that outlast your memory of the last hour. Keep one running document with the symptom numbers, each hypothesis and its prediction, the query you ran and what came back, and each change with its timestamp. It stops two people from retesting a theory that lost, and it gives a person joining at minute 40 something better than a scrollback to read.
The theory that arrives with the ticket is the hard part to practice against. On Katabench, the incident katas attached to this post supply one along with a report and a log excerpt, and the visible tests pass because the bug shipped, so the evidence has to lead you to it before a hidden regression suite replays the incident.
Good raw material shortens all of this: structured logs make the tenant split a query instead of a parsing job, and distributed traces separate waiting from working without a custom table.
Practice what you just read
- Open in the editor: The Outage at 0% Errors
Algorithm Medium Free, no account needed
The Outage at 0% Errors
A route failed for 40 minutes while the dashboard showed 0% errors, so the alert never fired. Where did 412 failed requests go?
- Open in the editor: One Sale, Two Business Days
Algorithm Medium Free, no account needed
One Sale, Two Business Days
During a terminal rollout, the same moment in the same shop landed on two different business days, and evening takings started drifting into tomorrow.
More like this: C# coding challenges →