Skip to content
Katabench
Try free
8 min read The Katabench team

Debugging production issues: a method for the first hour

Debugging production issues starts with numbers. A simulated webhook outage traces to one tenant: 4.1% of attempts holding 91% of worker time.

The ticket says "webhooks are late." Someone in the channel mentions the deploy that went out this afternoon, somebody else opens the delivery worker's source, and a third person has started a rollback. Twenty minutes later three people hold three theories and nobody can say how late is late.

Debugging production issues goes badly in a predictable way: the first plausible story gets all the effort. The cure is a fixed order of work that makes a story earn its place. Each step below produces something the next step needs.

The scenario is invented and the numbers are real output: a small discrete-event simulation of a webhook service, queried in PostgreSQL. Ten workers pull from one first-in, first-out queue at about 25 deliveries per second, across 40 tenants.

Turn the report into numbers

A symptom you can act on names a quantity, a threshold, a start time and a scope. Start with the two things a delivery does, waiting and attempting, and look at more than the middle of each:

SQL
SELECT to_char(date_bin('5 minutes', enqueued_at, '2026-11-01'), 'HH24:MI') AS enqueued,
       count(*) AS deliveries,
       round(percentile_cont(0.95) WITHIN GROUP (ORDER BY extract(epoch FROM
         coalesce(started_at, '2026-11-01 14:30') - enqueued_at))::numeric, 1) AS p95_wait_s,
       round(percentile_cont(0.5) WITHIN GROUP (ORDER BY extract(epoch FROM
         finished_at - started_at))::numeric, 2) AS p50_attempt_s,
       round(percentile_cont(0.99) WITHIN GROUP (ORDER BY extract(epoch FROM
         finished_at - started_at))::numeric, 2) AS p99_attempt_s
FROM deliveries GROUP BY 1 ORDER BY 1;
Text
 enqueued | deliveries | p95_wait_s | p50_attempt_s | p99_attempt_s
----------+------------+------------+---------------+---------------
 14:00    |       7582 |        0.0 |          0.12 |          0.26
 14:05    |       7552 |        0.0 |          0.12 |          0.28
 14:10    |       7591 |      684.9 |          0.12 |         30.00
 14:15    |       7497 |      843.4 |          0.12 |         30.00
 14:20    |       7433 |      584.4 |               |
 14:25    |       7625 |      286.1 |               |

"Webhooks are late" has become a claim you can test: p95 queue wait went from zero to over ten minutes while arrivals did not change. The p99 column guards against a near miss. Read alone, the flat median says attempts are fine and sends you straight to the queue, but a flat median rules out only a broad slowdown, because a small tenant cannot move it. At the 99th percentile, attempts jumped from under 0.3 seconds to exactly 30 seconds in the same bucket as the wait. Two things changed at once, and any explanation has to cover both.

The last two rows look like recovery, and they are an artifact. At 14:30, 21,307 deliveries were still queued. A row enqueued at 14:25 can only have waited five minutes so far, and without the coalesce it would be missing from the percentile altogether. Those rows have no attempt columns for the same reason. Any metric computed only from finished work flatters an outage while it is happening, so ask what your dashboard does with the requests still in flight.

Shrink the window until the start is a timestamp

The boundary is your strongest clue, because something changed there. Google's SRE book makes the same point about recent changes: a system tends to keep behaving as it was until something pushes it, whether a config change or a shift in load. Rerun the query per minute around the edge:

Text
 enqueued | deliveries | p95_wait_s
----------+------------+------------
 14:08    |       1468 |       0.00
 14:09    |       1488 |       0.00
 14:10    |       1565 |     124.61
 14:11    |       1490 |     311.24

The first delivery to wait more than a second was enqueued at 14:10:07. The rolling deploy of release v84 started at 14:05:00, one worker every 90 seconds. That gap of five minutes does not clear the deploy, but it does make you want a better reason than "it was recent."

Now look for other events in the same window, and write down what you find along with the time you looked. In this data, the first delivery attempt to hit the 30-second timeout started at 14:10:02.

Write down predictions before running queries

A hypothesis earns attention by naming what you would see if it were false. "The deploy did it" names nothing. "Attempts on v84 workers are slower or fail more often than attempts on v83 workers in the same minutes" can lose. Writing the prediction first also protects you from reading a result generously after the fact. The SRE book lists the related traps, among them latching on to the causes of past problems and chasing correlations that are coincidence.

Rank candidates by how cheaply you can test them, not by how much you like them, since a quick elimination is worth more than a slow confirmation. A clear negative result counts as progress, which the same chapter argues at length.

Write the prediction before you run the query

Release v84 broke delivery

If true

Attempts on v84 workers are slower or fail more than attempts on v83 workers in the same minutes

Measured

Same 0.12 s median and same 4.5% timeouts

Ruled out

Customer endpoints are slow everywhere

If true

Attempt duration rises across tenants

Measured

39 tenants unchanged at a 0.12 s median

Ruled out

One tenant is holding the workers

If true

One tenant's share of slot time far exceeds its share of attempts

Measured

4.1% of attempts, 91% of slot time

Survives

Measurements from the simulated webhook service in this post. A hypothesis that survives still has to predict the result of a change before you believe it.

Read the evidence before the code

Source code tells you what could happen. Data tells you what did, and it narrows which code deserves your reading. Comparing the two releases over the minutes when both were running keeps time out of the comparison, an easy mistake here, because a naive split would compare v83's early, lightly queued attempts with v84's later ones:

SQL
SELECT release, count(*) AS attempts,
       round(percentile_cont(0.5) WITHIN GROUP (ORDER BY extract(epoch FROM
         finished_at - started_at))::numeric, 2) AS p50_attempt_s,
       round(100.0 * count(*) FILTER (WHERE outcome = 'timeout') / count(*), 1) AS timeout_pct
FROM deliveries
WHERE started_at >= '2026-11-01 14:12' AND started_at < '2026-11-01 14:16'
GROUP BY release ORDER BY release;
Text
 release | attempts | p50_attempt_s | timeout_pct
---------+----------+---------------+-------------
 v83     |      575 |          0.12 |         4.5
 v84     |      992 |          0.12 |         4.5

The release hypothesis is out, and the rollback in progress would have cost time for nothing. Splitting attempt time by tenant settles the second hypothesis and suggests the third:

Text
    who    | attempts | p50_attempt_s | p95_attempt_s
-----------+----------+---------------+---------------
 other 39  |     7640 |          0.12 |          0.22
 tenant-17 |      327 |         30.00 |         30.00

Brendan Gregg's USE method supplies the next question: for each resource, check utilization, saturation and errors. Worker slots are the resource here, and the queue is their saturation. Summing attempt duration per tenant since 14:12 shows who holds them:

Text
  tenant   | attempts | pct_of_attempts | slot_seconds | pct_of_slot_time
-----------+----------+-----------------+--------------+------------------
 tenant-17 |      327 |             4.1 |         9810 |             91.0
 tenant-07 |      226 |             2.8 |           29 |              0.3

tenant-17 made 4.1% of the attempts and held 91% of the worker time, while the median attempt stayed at 0.12 seconds.

The arithmetic agrees with the query. tenant-17 sends about one delivery per second, and each one now holds a worker for 30 seconds, so it demands 30 busy workers from a pool of 10. Every other tenant's deliveries wait behind those hanging attempts in one shared line, which is why a failure at one customer's endpoint looks like an outage for all 40. It also explains the p99 jump: at about 4% of attempts, this tenant fills everything above the 96th percentile and never reaches the median.

Test the explanation by changing something

A story that fits the past can still be wrong, so make it predict a result you have not seen. If slot starvation is the cause, then capping each tenant at two concurrent attempts should bring other tenants back to their old wait times while tenant-17's own queue grows. The same simulation with that cap:

At 14:30 One shared queue Two slots per tenant
Other tenants' deliveries still queued 20,424 0
tenant-17 deliveries still queued 883 1,170
Other tenants' longest wait since 14:10 851.5 s 0.172 s

The prediction held, including the part that costs something: tenant-17's backlog grew. In a live incident this step is your mitigation as well, and you should apply the smallest reversible version first. A circuit breaker and a shorter timeout are the natural follow-ups, and queue-based load leveling explains why a shared queue moves pressure around without removing it.

Keep a timeline while the incident is running

The method depends on notes that outlast your memory of the last hour. Keep one running document with the symptom numbers, each hypothesis and its prediction, the query you ran and what came back, and each change with its timestamp. It stops two people from retesting a theory that lost, and it gives a person joining at minute 40 something better than a scrollback to read.

The theory that arrives with the ticket is the hard part to practice against. On Katabench, the incident katas attached to this post supply one along with a report and a log excerpt, and the visible tests pass because the bug shipped, so the evidence has to lead you to it before a hidden regression suite replays the incident.

Good raw material shortens all of this: structured logs make the tenant split a query instead of a parsing job, and distributed traces separate waiting from working without a custom table.

Practice what you just read

More like this: C# coding challenges →

Get new katas and .NET tips in your inbox

A short note when fresh kata land, plus the C# and performance tricks behind the grading. No spam, unsubscribe anytime.