The Outage at 0% Errors
Medium About 8 min Free, no account needed
A route failed for 40 minutes while the dashboard showed 0% errors, so the alert never fired. Where did 412 failed requests go?
The brief
Reported by: Customer support, after the fact
Impact: /export returned 500s for 40 minutes on Tuesday. That was 412 failed requests, and the alert never fired because the dashboard showed the route at 0% errors throughout.
Code: Aggregate in the metrics job
The gateway knew about every failure. The metrics job that feeds the dashboard reads that same access log and reported far less.
gateway 14:02:11 GET /export 200 310ms
gateway 14:02:40 GET /export 500 1204ms
gateway 14:02:41 GET /export 500 1198ms
... 410 more 500s on /export ...
gateway 14:41:58 GET /search 500 840ms
gateway 14:42:03 GET /search 200 95ms
... 94 more 200s on /search ...
metrics 14:45:00 route=/orders requests=18234 errors=2 totalMs=912044
metrics 14:45:00 route=/export requests=1 errors=0 totalMs=310
metrics 14:45:00 route=/search requests=1 errors=1 totalMs=840
alerts 14:45:00 route=/export errorRate=0.00 threshold=0.05 ok
alerts 14:45:00 route=/search errorRate=1.00 threshold=0.05 PAGE- The incident channel settled early on the gateway sampling low-traffic routes.
/ordersand the other routes inhotRoutesmatch the gateway exactly.- Low-traffic routes have been reading either 0% or 100% errors for weeks. Everyone had put that down to small samples.
- Release 4.12, three weeks ago, is the only deploy in that time. Its note reads "route table can now grow past 4,000 routes, no behavior change".
Fix Aggregate so every route's counts match the access log. Keep the output order and format as they are.
The visible tests all pass, because this bug shipped. A hidden regression suite replays the incident and its neighbours. Custom input replays a log line, and anything you write with Console.WriteLine shows up in the results.
Starter code
using System;
using System.Collections.Generic;
using System.Linq;
public class Solution
{
// hotRoutes: known up front, always reported, in this order, even with no traffic.
// requests: "ROUTE STATUS MILLISECONDS", oldest first. A status of 500 or above is an error.
// Returns "ROUTE REQUESTS ERRORS TOTAL_MS": hot routes first, then the rest in order of first request.
public string[] Aggregate(string[] hotRoutes, string[] requests)
{
var hot = new RouteStats[hotRoutes.Length];
for (var i = 0; i < hotRoutes.Length; i++)
{
hot[i] = new RouteStats(hotRoutes[i], 0, 0, 0);
}
var others = new List<RouteStats>();
var otherSlots = new Dictionary<string, int>();
foreach (var request in requests)
{
var parts = request.Split(' ');
var route = parts[0];
var failed = int.Parse(parts[1]) >= 500;
var milliseconds = long.Parse(parts[2]);
var slot = Array.IndexOf(hotRoutes, route);
if (slot >= 0)
{
hot[slot].Add(failed, milliseconds);
}
else if (otherSlots.TryGetValue(route, out slot))
{
others[slot].Add(failed, milliseconds);
}
else
{
otherSlots[route] = others.Count;
others.Add(new RouteStats(route, 1, failed ? 1 : 0, milliseconds));
}
}
return hot
.Concat(others)
.Select(stats => $"{stats.Route} {stats.Requests} {stats.Errors} {stats.TotalMs}")
.ToArray();
}
}
public record struct RouteStats(string Route, int Requests, int Errors, long TotalMs)
{
public void Add(bool failed, long milliseconds)
{
Requests++;
Errors += failed ? 1 : 0;
TotalMs += milliseconds;
}
}How it is graded
Visible examples help you iterate. Hidden tests then cover edge cases and larger inputs, and every test reports server-measured time and allocated bytes.
Related guides
What it practices
- Debugging from symptoms
- Work from a production symptom and a log excerpt back to the cause, then fix it without breaking what already works.
Related katas
- Reminders for Collected Parcels Medium Free Customers keep getting nightly reminders for parcels they collected days ago. Trace the failed pickup through the locker logs and fix it.
- One Sale, Two Business Days Medium Free During a terminal rollout, the same moment in the same shop landed on two different business days, and evening takings started drifting into tomorrow.
- Three Extra Ledger Entries Easy Free Big payout runs reach the bank with three rows too many, and always the same three. Find out why before the next run settles.
- Two Amounts of Tax on One Line Easy Free A customer rejected an invoice whose only line disagrees with its own total by a cent. How does one number come out as two?