Design a notification system: fan-out, retries, and consent

The first version of every notification system is three lines inside a business handler:

await email.SendAsync(order.CustomerEmail, "Order confirmed", body, cancellationToken);
await push.SendAsync(order.CustomerId, "Order confirmed", cancellationToken);
await sms.SendAsync(order.Phone, "Your order is confirmed", cancellationToken);

It works until the email provider has a slow afternoon and checkout starts timing out, because placing an order now waits on a third party's mail relay. Then a retry sends a one-time passcode twice. Then a marketing campaign of two million emails queues in front of the passcodes, and login breaks for forty minutes. Then someone who unsubscribed last week gets the campaign anyway, at three in the morning in their time zone, and the complaint reaches the legal team before it reaches you.

None of those failures is exotic. Designing a notification system is a fundamentals question with a specific shape: one event, many channels, many providers, and users who said no.

Accept, store, acknowledge

The producer, whether that is the checkout service or a campaign tool, should not know about providers at all. It posts a notification request to an API that validates it, stores it durably, and returns 202 Accepted with a status URL. The request carries an idempotency key chosen by the caller, so a retried POST from a nervous client creates one notification, not two:

public sealed record NotificationRequest(
    string IdempotencyKey,
    Guid UserId,
    string Template,
    JsonDocument Payload,
    NotificationPriority Priority);

public enum NotificationPriority
{
    Otp = 0,
    Transactional = 1,
    Bulk = 2,
}

The priority is not decoration. It decides which lane the work enters later, and it is far easier to demand it at the door than to infer it from a template name during an incident.

Consent is a step, not a filter at the end

Preferences are the part most designs bolt on last and the part that does the most damage when wrong. Treat the consent check as a first-class stage that runs before any fan-out, because it is cheapest there and because its decision is final. It answers, per user and per channel: has this person opted out of this channel or this category; are they inside quiet hours in their own time zone, in which case the message is delayed rather than dropped; have they already received today's cap of marketing messages; does the channel have a verified address at all.

Record the decision as a delivery row with a suppressed status and a reason. Support will ask why a customer did not get the email, and "the consent gate dropped it because of a hard bounce on 2026-08-30" is an answer; an absent row is not. Provider feedback flows back into the same store: a hard bounce or a spam complaint suppresses the channel for future sends, which is both a courtesy and a deliverability requirement.

Fan out through a topic into a queue per channel

After the gate, the notification is published to a topic, and each channel has its own subscription feeding its own queue. That is the ordinary topic versus queue split: fan-out between channels, competing workers within one channel. What makes it a notification platform rather than a generic pipeline is the isolation rule that comes with it.

Each channel gets its own queue, its own worker pool, and its own circuit breaker around its provider. When the email provider goes down, the email workers trip their breaker, the email queue grows, and push and SMS notice nothing, because they share no queue, no worker, and no connection pool with email. Put all three channels on one queue with one pool and a dead email provider blocks every worker on timeouts while push notifications sit behind them.

Priority is the second isolation rule. Inside each channel, a one-time passcode must never queue behind a campaign, so either run separate queues per priority per channel with workers that drain the passcode queue first, or use a broker with real priority support and enforce it. A campaign of two million messages is a two-million-message queue; a passcode is a thirty-second promise. They cannot share a line.

A notification platform is judged on the message it did not send twice, the message it did not send at all, and the passcode that did not wait behind a campaign.

One delivery attempt at a time

Retries are how every send becomes a duplicate. The worker times out on the provider call, the message is redelivered, and the customer gets two passcodes or two "your order shipped" pushes. The defence is the same one idempotency keys use for payments: a table with a unique constraint, written before the side effect, not after.

CREATE TABLE notifications (
    id uuid PRIMARY KEY,
    idempotency_key text NOT NULL UNIQUE,
    user_id uuid NOT NULL,
    template text NOT NULL,
    payload jsonb NOT NULL,
    priority smallint NOT NULL,
    created_at timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE deliveries (
    notification_id uuid NOT NULL REFERENCES notifications (id),
    channel text NOT NULL,
    status text NOT NULL,
    attempts int NOT NULL DEFAULT 0,
    lease_until timestamptz NOT NULL DEFAULT now(),
    provider_message_id text,
    last_error text,
    updated_at timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (notification_id, channel)
);

The primary key on (notification_id, channel) is the unique constraint that makes a second attempt visible. The worker claims the row first and sends second:

public async Task<bool> TryClaimAsync(
    Guid notificationId,
    string channel,
    CancellationToken cancellationToken)
{
    var leaseUntil = DateTimeOffset.UtcNow.AddMinutes(1);

    var claimed = await _db.Database.ExecuteSqlAsync(
        $"""
        INSERT INTO deliveries (notification_id, channel, status, attempts, lease_until)
        VALUES ({notificationId}, {channel}, 'sending', 1, {leaseUntil})
        ON CONFLICT (notification_id, channel) DO UPDATE
            SET status = 'sending',
                attempts = deliveries.attempts + 1,
                lease_until = EXCLUDED.lease_until
            WHERE deliveries.status IN ('sending', 'failed')
              AND deliveries.lease_until < now()
        """,
        cancellationToken);

    return claimed == 1;
}

Two workers holding the same redelivered message both run this statement; one inserts and the other conflicts, finds a live lease, updates nothing, and drops the message. If the winner crashes mid-send, the lease expires and the next redelivery claims the row again with attempts incremented. A failed send marks the row failed and pushes lease_until out by the backoff, so the same column doubles as the not-before time, and after a bounded number of attempts the row moves to a dead-letter lane instead of retrying forever. Pass notification_id:channel to the provider as its idempotency key where it supports one; where it does not, a send that timed out after reaching the provider is the one duplicate this design cannot rule out. Decide per priority which failure you prefer: a duplicate passcode is annoying, a missing one locks someone out, and the opposite holds for a campaign.

Delivery receipts close the loop. Providers report delivered, bounced, and complained through webhooks, and those events update the delivery row and feed the consent store. A send the provider accepted is not a send the user received.

The campaign arithmetic

A campaign to 10 million users that must finish inside 30 minutes is a throughput requirement:

10,000,000 sends / 1,800 s = 5,556 sends per second, call it 5,600
provider allows 1,000 requests/s: 10,000,000 / 1,000 = 10,000 s = 2.8 hours

If the provider caps you at a fifth of the rate you need, no worker count fixes it; the options are a higher contracted limit, a second provider sharing the load, or a campaign window that is honest about the number. Assume the limit is negotiated up. With a 200 millisecond provider round trip, one in-flight send completes five per second, so 5,600 per second needs about 1,120 concurrent sends, which is fourteen workers at 80 in-flight each, or twenty-three at 50. The delivery table takes the same 5,600 claim writes per second plus the status updates, which is the number to check against the database before trusting the worker count.

The metrics that say the platform is healthy are per channel and per provider: delivery rate, bounce and complaint rate, breaker state, and the age of the oldest pending message in each lane. Depth alone lies, for the reasons the guide on queue-based load leveling works through: the backlog can be falling while the oldest passcode is still getting older.

Build it on the canvas before you build it in code

The individual pieces here are ordinary. What separates a notification platform that survives a provider outage from one that takes checkout down with it is the topology: where the consent gate sits, one queue per channel or one for all, whether the passcode lane exists at all. Katabench's System Design Studio has a Notification Platform build course that grows this design step by step on a canvas, with authored rules that check the fan-out, the per-channel isolation, and the forbidden bypasses, and a deterministic capacity simulation that reports which component saturates first under a campaign-sized traffic scenario. The Production Outbox Lab covers the other half, where a worker really crashes mid-batch against PostgreSQL and RabbitMQ and you measure what gets redelivered. The design is the easy part to draw and the hard part to size, which is why the arithmetic above is worth doing before the diagram.