Skip to content
Katabench
Try free

The Outage at 0% Errors

Medium About 8 min Free, no account needed

A route failed for 40 minutes while the dashboard showed 0% errors, so the alert never fired. Where did 412 failed requests go?

The brief

Reported by: Customer support, after the fact Impact: /export returned 500s for 40 minutes on Tuesday. That was 412 failed requests, and the alert never fired because the dashboard showed the route at 0% errors throughout. Code: Aggregate in the metrics job

What on-call found

The gateway knew about every failure. The metrics job that feeds the dashboard reads that same access log and reported far less.

Log excerpt

Text
gateway 14:02:11 GET /export 200 310ms
gateway 14:02:40 GET /export 500 1204ms
gateway 14:02:41 GET /export 500 1198ms
        ... 410 more 500s on /export ...
gateway 14:41:58 GET /search 500 840ms
gateway 14:42:03 GET /search 200 95ms
        ... 94 more 200s on /search ...
metrics 14:45:00 route=/orders requests=18234 errors=2 totalMs=912044
metrics 14:45:00 route=/export requests=1 errors=0 totalMs=310
metrics 14:45:00 route=/search requests=1 errors=1 totalMs=840
alerts  14:45:00 route=/export errorRate=0.00 threshold=0.05 ok
alerts  14:45:00 route=/search errorRate=1.00 threshold=0.05 PAGE

What we know so far

  • The incident channel settled early on the gateway sampling low-traffic routes.
  • /orders and the other routes in hotRoutes match the gateway exactly.
  • Low-traffic routes have been reading either 0% or 100% errors for weeks. Everyone had put that down to small samples.
  • Release 4.12, three weeks ago, is the only deploy in that time. Its note reads "route table can now grow past 4,000 routes, no behavior change".

Your job

Fix Aggregate so every route's counts match the access log. Keep the output order and format as they are.

Working the incident

The visible tests all pass, because this bug shipped. A hidden regression suite replays the incident and its neighbours. Custom input replays a log line, and anything you write with Console.WriteLine shows up in the results.

Starter code

C#
using System;
using System.Collections.Generic;
using System.Linq;

public class Solution
{
    // hotRoutes: known up front, always reported, in this order, even with no traffic.
    // requests: "ROUTE STATUS MILLISECONDS", oldest first. A status of 500 or above is an error.
    // Returns "ROUTE REQUESTS ERRORS TOTAL_MS": hot routes first, then the rest in order of first request.
    public string[] Aggregate(string[] hotRoutes, string[] requests)
    {
        var hot = new RouteStats[hotRoutes.Length];
        for (var i = 0; i < hotRoutes.Length; i++)
        {
            hot[i] = new RouteStats(hotRoutes[i], 0, 0, 0);
        }

        var others = new List<RouteStats>();
        var otherSlots = new Dictionary<string, int>();

        foreach (var request in requests)
        {
            var parts = request.Split(' ');
            var route = parts[0];
            var failed = int.Parse(parts[1]) >= 500;
            var milliseconds = long.Parse(parts[2]);

            var slot = Array.IndexOf(hotRoutes, route);
            if (slot >= 0)
            {
                hot[slot].Add(failed, milliseconds);
            }
            else if (otherSlots.TryGetValue(route, out slot))
            {
                others[slot].Add(failed, milliseconds);
            }
            else
            {
                otherSlots[route] = others.Count;
                others.Add(new RouteStats(route, 1, failed ? 1 : 0, milliseconds));
            }
        }

        return hot
            .Concat(others)
            .Select(stats => $"{stats.Route} {stats.Requests} {stats.Errors} {stats.TotalMs}")
            .ToArray();
    }
}

public record struct RouteStats(string Route, int Requests, int Errors, long TotalMs)
{
    public void Add(bool failed, long milliseconds)
    {
        Requests++;
        Errors += failed ? 1 : 0;
        TotalMs += milliseconds;
    }
}

How it is graded

Visible examples help you iterate. Hidden tests then cover edge cases and larger inputs, and every test reports server-measured time and allocated bytes.

C# coding challenges →

Related guides

What it practices

Debugging from symptoms
Work from a production symptom and a log excerpt back to the cause, then fix it without breaking what already works.

Related katas