Design a Notification System: Queues, Retries and Fan-out

Design a notification system looks simple. Send an email, a push, an SMS. Done.

Then the interviewer asks: What if the email provider is down? What if a user gets the same push twice? What if you need to notify 5 million followers at once? That is where the real design starts.

This post walks through a clean answer to notification system design, step by step.

1. Clarify requirements first

Ask before you draw:

• Which channels? Push, email, SMS, in-app.

• Real-time or can it be delayed?

• Do users have preferences (mute, quiet hours)?

• Scale: notifications per day, peak bursts.

Non-functional goals: no lost notifications, no duplicates, low latency for urgent ones (OTP), and one slow channel must not block the others.

2. High-level architecture
Service A ─┐
Service B ─┼─► Notification API ─► Validate + Preferences ─► Message Queue(s)
Service C ─┘ │
┌───────────────────────┼──────────────┐
▼ ▼ ▼
Push Worker Email Worker SMS Worker
│ │ │
APNs/FCM Email provider SMS provider

Key decisions:

• Decouple with a queue. Callers get an instant "accepted" response. Spikes don't hit providers directly.

• Separate queue per channel (and per priority). An SMS outage should not delay emails.

• Stateless workers so you can scale each channel independently.

3. Fan-out: one event, millions of users

When one post notifies 5 million followers, don't loop inside the API request. Publish one event, then let a fan-out service expand it into per-user messages in batches.

Fan-out on write: create per-user notifications immediately. Fast reads, heavy writes.
Fan-out on read: compute when the user opens the app. Cheap writes, slower reads.

4. Retries with exponential backoff

Providers fail. A good worker retries, but not blindly.

import random
import time

MAX_RETRIES = 5

def send_with_retry(message):
    for attempt in range(MAX_RETRIES):
        try:
            provider.send(message)
            return "sent"
        except TemporaryError:
            if attempt == MAX_RETRIES - 1:
                dead_letter_queue.publish(message)
                return "failed"

            delay = min(30, 2 ** attempt)
            time.sleep(random.uniform(0, delay))

Three things to say out loud:

• Backoff stops you from hammering a struggling provider.

• Jitter stops all workers retrying at the same second.

• Dead-letter queue (DLQ) keeps failed messages for inspection and replay.

5. Avoiding duplicates

Queues usually guarantee at-least-once delivery, so duplicates will happen. Fix it with an idempotency key:

def handle(message):
    key = f"sent:{message.notification_id}:{message.channel}"

    # Check if already processed
    if redis.get(key):
        return "already_processed"

    # Send notification
    result = send_with_retry(message)

    # Mark as processed after successful sending
    if result == "sent":
        redis.set(key, "1", ex=86400)

    return result

Same notification ID, same channel, processed once.

6. Preferences, rate limits and priority

• Preferences check before queuing: opted out, quiet hours, channel choice.

• Per-user rate limit so a bug upstream doesn't spam someone with 200 pushes.

• Priority queues: OTP and security alerts must never wait behind marketing campaigns.

7. Tracking status

Store each notification with a status: queued → sent → delivered → opened / failed. Providers send delivery callbacks (webhooks) that update this table. It powers analytics and debugging.

Interview checklist

• Clarified channels, scale and latency needs

• Queue between API and workers

• Separate queues per channel and priority

• Retries with backoff, jitter and a DLQ

• Idempotency key for deduplication

• Preferences and rate limiting

• Delivery tracking

Practice it for real

Knowing the pieces is one thing. Explaining the trade-offs under a timer is another. If you want hands-on system design practice instead of just reading, ScaleDojo has 220+ interactive labs across HLD, LLD, API design and GenAI/RAG, so you can build designs like this and get feedback.

What would you add to this design? Drop it in the comments.