Medium

Design a Notification System

Design a platform service that delivers notifications to users across push (iOS/Android), email, and SMS on behalf of many internal product teams. The interesting problems are fan-out at scale, deduplication, per-user rate limiting and preferences, and reliable integration with flaky third-party providers like APNs, FCM, and SMS gateways.

1Requirements

Functional

• Internal services can send a notification to one user, a list of users, or a segment (e.g., all users in a city) via a single API.
• Support push notification, email, and SMS channels, with per-notification channel selection and fallbacks (e.g., SMS if push unread after 30 min).
• Users can set preferences and opt-outs per channel and per notification category (marketing vs. transactional).
• Deduplicate notifications so a user never receives the same logical event twice, even under producer retries.
• Support scheduled and delayed delivery (e.g., send at 9am in the user's timezone).
• Track delivery status (sent, delivered, failed, opened) and expose it to producing teams.

Non-functional

• At-least-once delivery into the pipeline with dedup at the edge, so users effectively see at-most-once per event.
• Soft real-time: transactional notifications (OTP, payment alerts) delivered within seconds; bulk campaigns can take minutes.
• Scale to ~1 billion notifications/day across all channels.
• Rate limit per user (avoid spamming) and per provider (respect APNs/FCM/SMS gateway quotas).
• High availability: the ingestion API must accept sends even when downstream providers are degraded.

2Back-of-envelope estimation

Notifications per day	1 billion	~800M push, 150M email, 50M SMS; push dominates because it is cheap.
Average / peak send rate	~12K/s avg, ~120K/s peak	1B / 86,400s ≈ 11.6K/s; marketing campaigns create 10x bursts, which is exactly why queues sit in the middle.
SMS cost	~$375K/day	50M SMS x ~$0.0075 each; cost alone justifies aggressive channel fallback ordering (push first, SMS last).
Device token storage	~150 GB	500M users x ~2 devices x ~150 bytes per token record.
Delivery log storage	~500 GB/day raw	1B events x ~500 bytes; keep 30 days hot (~15 TB), archive the rest to cold storage.

3API design

POST /v1/notifications

Producer API: accepts {idempotency_key, recipient(s) or segment_id, category, channels, template_id, payload, schedule_time}. Returns 202 with a notification_id immediately; delivery is async.

GET /v1/notifications/{id}/status

Returns per-recipient, per-channel delivery status (queued, sent, delivered, failed, opened) for producer teams and support tooling.

PUT /v1/users/{userId}/preferences

Sets per-channel, per-category opt-in/opt-out and quiet hours; enforced centrally so every producing team gets compliance for free.

POST /v1/devices

Registers or refreshes a device push token (APNs/FCM) for a user; called by mobile clients on app start and token rotation.

4High-level design

Producing services call the notification API with an idempotency key. The API validates the request, checks the key against a dedup store (Redis with TTL, backed by a persistent table), persists the notification record, and drops a message onto a Kafka topic. Returning 202 here decouples producer latency from provider latency entirely.

A fan-out service consumes the topic. For a single recipient it is a passthrough; for a segment it queries the user service to expand membership into batches of individual sends. Fan-out output is written to per-channel queues (push, email, SMS), which lets each channel scale, throttle, and fail independently.

Before enqueueing per-channel work, a preference-and-policy layer runs: check the user's opt-outs for the category, apply quiet hours and timezone scheduling, run per-user rate limiting (e.g., max 5 marketing pushes/day via a Redis token bucket), and select the channel per the fallback policy.

Channel workers pull from their queue and call the third-party provider: APNs and FCM for push, an ESP like SES/SendGrid for email, Twilio or a direct carrier gateway for SMS. Workers batch where providers allow it, apply per-provider rate limits, and retry with exponential backoff on transient failures; poison messages go to a dead-letter queue for inspection.

Provider callbacks and receipts (APNs feedback, ESP webhooks, SMS DLRs) flow into a delivery-tracking service that updates the notification status store and prunes invalid device tokens. Analytics jobs aggregate open and failure rates per template and per provider, which feeds alerting and provider failover decisions.

5Data model

notification

id BIGINT PK, idempotency_key VARCHAR UNIQUE, producer_id VARCHAR, category VARCHAR, template_id VARCHAR, payload JSONB, schedule_time TIMESTAMP, created_at TIMESTAMP

One row per logical send request; the unique idempotency key is the dedup backstop behind the Redis cache.

delivery

id BIGINT PK, notification_id BIGINT FK, user_id BIGINT, channel VARCHAR, provider VARCHAR, status VARCHAR, provider_message_id VARCHAR, updated_at TIMESTAMP

One row per recipient per channel attempt; partitioned by time, this is the highest-volume table.

device_token

user_id BIGINT, token VARCHAR, platform VARCHAR, app_version VARCHAR, last_active TIMESTAMP, valid BOOLEAN, PK (user_id, token)

Pruned when APNs/FCM report the token invalid; stale tokens are the top cause of push 'failures'.

user_preference

user_id BIGINT, category VARCHAR, channel VARCHAR, opted_in BOOLEAN, quiet_hours_start TIME, quiet_hours_end TIME, timezone VARCHAR, PK (user_id, category, channel)

6Deep dives

Deduplication and idempotency end to end

Duplicates enter from two directions: producers retrying the API call, and the pipeline redelivering messages (Kafka consumers are at-least-once). Producer-side duplicates are handled by requiring an idempotency key per logical event (e.g., 'order-1234-shipped'); the API checks Redis first and falls back to a unique constraint on the notifications table, so a crashed Redis never lets a duplicate through.

Pipeline-side duplicates need a second check close to the send: before a channel worker calls the provider, it does a conditional write on the delivery row (status queued -> sending). If the row is already in sending/sent, another worker won the race and this attempt is dropped. This is a classic transactional outbox pattern in reverse: the state transition in the DB is the source of truth for whether a send may happen.

Note what this does not solve: if the worker crashes after calling APNs but before recording 'sent', a retry can still double-send. True exactly-once to an external provider is impossible; you minimize the window by recording the attempt before the provider call and treating an ambiguous outcome as sent for user-facing notifications (a missed notification is usually worse than nothing, but a duplicate OTP is harmless while a duplicate marketing push is annoying, so per-category policy applies).

Rate limiting: users, providers, and campaigns

Three distinct rate limits coexist. Per-user limits protect the user experience: a Redis token bucket keyed by (user_id, category) caps marketing sends per day while exempting transactional messages like OTPs. Enforcing this centrally in the policy layer is a major selling point of a shared platform, since no individual product team can spam users past the global cap.

Per-provider limits protect your standing with APNs, FCM, ESPs, and carriers. Channel workers share a distributed rate limiter per provider connection pool; exceeding SMS carrier throughput, for instance, gets messages silently queued or dropped by the carrier. Provider limits also drive worker autoscaling: there is no point scaling SMS workers past the gateway's throughput ceiling.

Third-party provider integration and failover

APNs uses HTTP/2 with long-lived connections and token-based (JWT) auth; FCM has its own HTTP v1 API. Both return per-message errors that must be interpreted: an 'Unregistered'/'BadDeviceToken' response means the token is dead and must be pruned, while 5xx or throttling responses mean back off and retry. Conflating the two either spams dead tokens forever or drops live users.

For email and SMS, run at least two providers with weighted routing and health-based failover: if SendGrid's error rate spikes, shift traffic to SES. The abstraction that makes this clean is a provider-agnostic send interface per channel with adapters per vendor, plus delivery-receipt normalization so downstream tracking does not care which vendor sent the message.

Webhooks from providers (bounces, complaints, DLRs) are ingested through a public callback endpoint into the same Kafka backbone. Email bounce and complaint handling is not optional: ESPs will suspend accounts with high complaint rates, so hard bounces must automatically suppress the address.

7Rapid implementation: build the MVP

Theory is table stakes. Here is how you would stand up a working version fast, on a budget, with the core algorithm in real code.

Stack

Node.js + Postgres as a transactional outbox, Redis for rate caps, Expo push + Resend email in free tiers, one Fly.io machine.

01Create Postgres tables outbox and user_preferences; the unique index on idempotency_key is your producer-side dedup.
02Build POST /notifications: INSERT into outbox with ON CONFLICT (idempotency_key) DO NOTHING and return 202 with the row id either way.
03Write the worker loop: every second, claim a batch of due rows with FOR UPDATE SKIP LOCKED so concurrent workers never grab the same row.
04In the worker, check the user's opt-outs and quiet hours, then apply a per-user daily marketing cap via a Redis counter with a 24h TTL.
05Wire two providers behind a common send(userId, payload) interface: Expo for push, Resend for email; add SMS later only if you must pay for it.
06On provider failure, bump attempts and push send_after forward with exponential backoff; after 5 attempts mark the row dead for inspection.
07Expose GET /notifications/:id/status reading straight off the outbox row, and a webhook endpoint that records provider delivery receipts.

Outbox table and atomic batch claim

sql

CREATE TABLE outbox (
  id BIGSERIAL PRIMARY KEY,
  idempotency_key TEXT UNIQUE NOT NULL,
  user_id BIGINT NOT NULL,
  channel TEXT NOT NULL,          -- 'push' | 'email' | 'sms'
  category TEXT NOT NULL,         -- 'transactional' | 'marketing'
  payload JSONB NOT NULL,
  status TEXT NOT NULL DEFAULT 'queued',
  attempts INT NOT NULL DEFAULT 0,
  send_after TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX ON outbox (status, send_after);

-- Worker claims a batch atomically; SKIP LOCKED means
-- concurrent workers never double-send the same row.
UPDATE outbox SET status = 'sending', attempts = attempts + 1
WHERE id IN (
  SELECT id FROM outbox
  WHERE status = 'queued' AND send_after <= now()
  ORDER BY id LIMIT 100
  FOR UPDATE SKIP LOCKED
)
RETURNING *;

Worker with per-user rate cap and backoff

typescript

const DAILY_MARKETING_CAP = 5;

async function processBatch(rows: OutboxRow[]) {
  for (const row of rows) {
    // transactional messages (OTP, receipts) are exempt from caps
    if (row.category === "marketing") {
      const key = "cap:" + row.user_id + ":" + isoDate();
      const n = await redis.incr(key);
      if (n === 1) await redis.expire(key, 86400);
      if (n > DAILY_MARKETING_CAP) {
        await setStatus(row.id, "suppressed");
        continue;
      }
    }
    try {
      await providers[row.channel].send(row.user_id, row.payload);
      await setStatus(row.id, "sent");
    } catch {
      if (row.attempts >= 5) {
        await setStatus(row.id, "dead"); // poison row, inspect later
      } else {
        const delayMs = Math.min(2 ** row.attempts * 1000, 3600_000);
        await requeue(row.id, delayMs); // status back to queued
      }
    }
  }
}

Bottlenecks & failure modes

⚠Segment fan-out: expanding 'all users in California' into 20M individual sends can flood the pipeline; batch the expansion and rate-limit campaign injection so transactional traffic keeps priority.
⚠Third-party provider throttling or outages; mitigated by per-provider queues, backoff, and multi-provider failover for email/SMS.
⚠The delivery status table grows by ~1B rows/day; requires time-based partitioning, async writes, and tiered retention.
⚠Hot users (e.g., a celebrity's followers all notified at once) are fine, but hot producers misconfiguring a loop can self-DDoS the platform; per-producer quotas at the API are essential.
⚠Priority inversion: bulk marketing campaigns queued ahead of OTPs; solve with separate priority queues or topics per traffic class.

Key takeaways

▸Accept fast, deliver async: a 202 plus a queue decouples producer latency from the slowest SMS gateway.
▸Per-channel queues and workers let push, email, and SMS scale and fail independently under one API.
▸Idempotency keys at the API plus conditional state transitions at the send step give effective at-most-once user experience over an at-least-once pipeline.
▸Centralized preferences, quiet hours, and per-user rate limits are the real product of a notification platform, not just message plumbing.
▸Treat providers as unreliable dependencies: normalize their errors, prune dead tokens, and keep a second vendor warm.

Brush up on the underlying topics

Message Queues and Streaming Rate Limiting Event-Driven Architecture Fault Tolerance Patterns API Design