Queue-based load leveling: absorb write spikes without losing work

The import endpoint has been fine for a year. Then a partner schedules a bulk sync for nine in the morning, and for two minutes the API takes 5,000 writes per second instead of the usual few hundred. Every request does the same five things before it answers: validate, insert, update the search index, send a confirmation, record analytics. At 5,000 per second the connection pool is gone in the first few seconds, latency climbs past the client timeout, the client retries, and now the spike is bigger than it was. The database never failed. The request path did, because it insisted on finishing all five things while the caller waited.

Queue-based load leveling is the boring fix. The request path does only what must happen before the response and hands the rest to a durable queue that workers drain at a rate you choose. The arrival rate and the processing rate stop being the same number. That decoupling is the whole pattern, and it has a price.

Only do what must happen before the response

For a bulk import, the caller needs an identifier and a promise. It does not need the search index updated before it hears back. So the endpoint records the intent, enqueues a message, and returns 202 Accepted with a status resource the caller can poll:

app.MapPost("/imports", async (
    ImportRequest request,
    ImportsDb db,
    IWorkQueue queue,
    CancellationToken cancellationToken) =>
{
    var import = new Import(Guid.NewGuid(), request.FileKey, ImportStatus.Queued);
    db.Imports.Add(import);
    await db.SaveChangesAsync(cancellationToken);

    await queue.EnqueueAsync(new ImportQueued(import.Id), cancellationToken);

    return Results.Accepted($"/imports/{import.Id}", new { import.Id, import.Status });
});

POST /imports HTTP/1.1
Content-Type: application/json

{ "fileKey": "partners/acme/2026-09-08.csv" }

HTTP/1.1 202 Accepted
Location: /imports/7f3c9a2e-5b1d-4c8e-9f0a-2d6b8e4c1a77
Content-Type: application/json

{ "id": "7f3c9a2e-5b1d-4c8e-9f0a-2d6b8e4c1a77", "status": "Queued" }

The save and the enqueue are still two writes, and a crash between them loses the message. That is the gap the transactional outbox closes; for an import with a status row the caller can re-submit, a retry is often enough. Either way the request now costs one insert and one enqueue, which a database and a broker can absorb at 5,000 per second without much drama. The spike still exists. It has moved into the queue.

Work the numbers before you trust the queue

A queue does not make work disappear. Say the spike is 5,000 writes per second for two minutes and the workers drain 1,000 per second:

arrivals during the spike:   5,000/s x 120 s = 600,000
drained during the spike:    1,000/s x 120 s = 120,000
backlog when the spike ends: 480,000
time to clear at 1,000/s:    480,000 / 1,000 = 480 s = 8 minutes

Little's law says the same thing from the other side: the average number of items in a stable system equals the arrival rate times the average time each item spends there, so a backlog of 480,000 in front of a drain of 1,000 per second means the message at the back waits 480 seconds. The last write of the spike is processed ten minutes after the first one arrived.

Whether eight minutes is fine is a product question, and it is the question the queue forces you to answer explicitly. For a search index update it probably is. For a confirmation the customer is staring at, it is not, and no amount of broker configuration changes that. If the usual few hundred writes per second keep arriving after the spike, the net drain is smaller and the clear time grows: at 400 per second of baseline traffic the queue empties at 600 per second, and 480,000 takes 800 seconds, closer to thirteen minutes.

A queue does not remove the wait. It moves the wait from the caller's open connection to the consumer's backlog, and it makes the drain rate a promise you now have to keep.

When the queue never drains

Leveling works because the spike ends. If arrivals stay above the drain rate, the backlog grows without bound: 1,200 per second in against 1,000 out adds 200 per second, which is 720,000 messages and another twelve minutes of wait for every hour it continues. The broker eventually runs out of disk or memory, consumers process messages whose deadline passed an hour ago, and the outage the queue was hiding arrives on a delay, larger than it would have been.

So the queue needs a bound, and the producer needs an answer for when the bound is hit. A bounded queue that rejects new messages turns into 429 Too Many Requests or 503 Service Unavailable at the API, with a Retry-After header so well-behaved clients back off instead of hammering. That is the same conversation as rate limiting at the gateway, and the same discipline: say no loudly and early rather than accepting work you cannot finish. If some messages matter more than others, give them separate queues and shed the cheap ones first. An analytics event dropped during a spike is a rounding error; an order dropped during a spike is a support ticket.

Acknowledge after the work is durable

The consumer is where most of the correctness lives. The rule is short: acknowledge a message only after its effect is durable. Acknowledge on receipt and a crash mid-handler silently loses the work; acknowledge after commit and a crash means the broker redelivers. Redelivery is the right failure, but it means every handler must tolerate seeing a message twice, which is the queue version of an idempotent API.

public sealed class ImportConsumer(
    IWorkQueue queue,
    IImportHandler handler,
    ILogger<ImportConsumer> logger) : BackgroundService
{
    private const int MaxAttempts = 5;
    private readonly SemaphoreSlim _slots = new(initialCount: 8, maxCount: 8);

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        await foreach (var message in queue.ReceiveAsync(stoppingToken))
        {
            await _slots.WaitAsync(stoppingToken);
            _ = ProcessAsync(message, stoppingToken);
        }
    }

    private async Task ProcessAsync(QueueMessage message, CancellationToken cancellationToken)
    {
        try
        {
            await handler.HandleAsync(message.Body, cancellationToken);
            await queue.AckAsync(message, cancellationToken);
        }
        catch (Exception exception) when (!cancellationToken.IsCancellationRequested)
        {
            if (message.DeliveryCount >= MaxAttempts)
            {
                logger.LogError(exception,
                    "Dead-lettering {MessageId} after {Attempts} attempts",
                    message.Id, message.DeliveryCount);
                await queue.DeadLetterAsync(message, exception.Message, cancellationToken);
            }
            else
            {
                await queue.NackAsync(message, requeue: true, cancellationToken);
            }
        }
        finally
        {
            _slots.Release();
        }
    }
}

Three decisions are encoded here. The semaphore bounds concurrency at eight in-flight messages per instance, so the worker cannot hand the database more parallel work than it was sized for. The acknowledgement follows the handler, not the receive. And a message that fails five times goes to a dead-letter queue instead of circling forever: a poison message, one with a malformed payload or a reference to a row that no longer exists, would otherwise occupy a slot on every redelivery and quietly eat the drain rate. Dead-lettered messages need an owner and an alert; a dead-letter queue nobody reads is a slower way of losing work. Real brokers add a prefetch count, a redelivery delay, and a graceful shutdown that waits for in-flight handlers, but the shape stays the same.

Watch the age, not the depth

Queue depth is the metric everyone graphs and the one that misleads. Go back to the arithmetic. The backlog peaks at 480,000 at the end of minute two and then falls steadily, so five minutes in the depth chart looks like a recovery in progress. The oldest message in the queue disagrees. The consumer is working through traffic that arrived at 5,000 per second at only 1,000 per second, so the front of the queue advances through arrival time at a fifth of the speed of the clock. The age of the oldest pending message rises by 48 seconds every minute and peaks at 480 seconds at the moment the queue finally empties. Depth says the incident is half over; age says the worst latency anyone will see is still ahead.

Alert on the age of the oldest message against the latency the work can tolerate, and scale the workers on that too. Sizing the pool is division, the same back-of-the-envelope arithmetic that sizes everything else: if a handler spends 40 milliseconds in the database, one worker with eight slots processes about 200 messages per second, so a drain of 1,000 per second takes five workers. Draining 480,000 in two minutes instead of eight would take 4,000 per second and twenty workers, and at that point the database is absorbing the spike again, just from the other side. Load leveling only protects a dependency if the drain rate stays below what that dependency can sustain. The queue chooses where the ceiling is applied; it does not raise it.

Practice the spike before it practices on you

The pattern is easy to describe and easy to get wrong at the numbers: a drain rate nobody sized, a queue nobody bounded, a dashboard showing depth while age climbs. Katabench's System Design Studio has a fundamentals challenge called Buffer the Write Spike where you put a queue and workers between the API and the database on a canvas, and a deterministic capacity simulation runs a spike scenario against it and reports p99 latency, throughput, and the component that saturates first. Size the workers too small and the model shows the backlog and the tail latency; size them too large and the database is the component that gives.

The consumer side is what the Production Outbox Lab is for: a polling outbox with PostgreSQL and RabbitMQ in a running workspace, where you crash the consumer mid-batch and measure what gets redelivered. If you want the delivery topology behind the queue first, the guide on message queues versus event buses is the place to start.