A System Designer guide
Notification service walkthrough
Separate acceptance from sending, respect provider quotas and reason about expiry and unknown outcomes before claiming delivery once.
This walkthrough uses the Notification service scenario. Try an attempt first if you want to make your own choices before reading a possible approach.
Requirements and scale
Every other team asks you to send messages: order shipped, new login, a sale starting. You send them by push, email and text through outside providers who have their own limits. Send each message once, and never block the team that asked.
- Who uses it
- 30 million people a month
- Audience in plain numbers
- 30,000,000 people per month
- Normal peak
- ≈ 8,000 requests per second
- Request mix
- Notification: 70%; Analytics event: 20%; Write: 10%
- Speed for 99 of 100 requests
- Write: ≈ 300 ms
- Successful requests
- 99.9%
- Data safety
- No acknowledged write may be lost.
- Monthly budget
- ≈ $120,000
- Data at rest
- ≈ 2 TB
- Where users are
- North America: 60%; Europe: 40%
- Home region
- North America
- Waiting deadlines
- Notification: 1,800 seconds; Analytics event: 1,800 seconds; Write: 1,800 seconds
- Required proofs
- Consent records
≈ marks an approximation, just as in the game. Waiting deadlines measure the age of queued work and are separate from interactive response time.
- Providers cap how fast you may send: push ≈ 10,000/s, email ≈ 1,000/s, text ≈ 100/s (approx.).
- Texts cost far more than email or push.
- Notification speed measures provider acceptance. Device receipt is outside this traffic model; check expiry, provider outcome and retries in the correctness trace before claiming delivery guarantees.
Estimates to work through
- Messages to send per second at peak
- ≈ 5,600 msg/s. 8,000 × 0.7
- Messages per day
- ≈ 190 million. Average load is about 0.4 of peak: 5,600 × 0.4 × 86,400 ≈ 190 million (approx.)
- Email sending limit
- ≈ 1,000 email/s. The provider cap in this scenario: you must spread load across channels and wait politely (approx.)
Acceptance is a promise too
The caller should not wait for an email or push provider to finish. It needs to know whether work was accepted durably. Give the request an identity, check the required preference policy and record the job before returning acceptance. Decide what the caller can safely retry if the reply is lost.
Separate urgent messages from campaigns. A message that can wait through a provider outage has a different deadline from a login code. The scenario’s waiting deadline and provider-acceptance boundary below are the promises to test; they do not prove device receipt.
A design that holds
Use durable queues to decouple callers from sending. Workers consume at a pace each provider allows, recheck preferences, track attempts and preserve the same logical key across retries. Keep an explicit state for work whose provider outcome is unknown.
A local duplicate lookup alone cannot guarantee one external send after a crash. If a provider accepted the message but the worker died before recording that fact, a retry needs the provider’s idempotency support or a reconciliation path. State that dependency instead of claiming a queue solves it.
Separate channels or urgency classes so one throttled provider cannot block everything. Quarantine permanently bad work with a dead-letter policy, meaning a place for failed jobs that need inspection rather than endless retries. Track oldest waiting age and terminal outcomes, not only queue length.
The trade-offs
Buffering absorbs bursts but spends the delivery deadline. More workers only help while the destination has capacity. More providers add resilience but also quotas, costs and an ambiguous handover point where both may send the same message.
Texts have a much smaller quota and a different bill from push or email in this scenario. Do not split traffic equally across channels without checking those limits. A cheap default channel is useful only if it meets the message’s product requirements.
What breaks it
Throttle a provider and keep accepting new work. Ask whether spare sending capacity can clear the backlog before the oldest jobs expire. Next make one message fail repeatedly and check that urgent work still progresses. Finally trace a lost provider reply and explain how the next worker discovers the outcome.
Use Waiting instead of breaking and Work across services, then try Notification service. Defend the difference between a fast acceptance response and a completed send using both the run and a crash trace.
The day that tests the design
Consumer traffic, busiest in the evening. The clock covers 00:00 to 24:00 in the home region. Events add pressure on top of that normal traffic.
09:00 · Provider rate limit
An outside service lowers its configured request quota and answers 'too many requests' above the reduced limit.
Provider capacity falls to 40% of its configured quota. Duration: ≈ 3,600 seconds.
13:00 · Duplicate delivery
The queue redelivers messages. Without duplicate protection, side effects may be applied twice.
Affected share: 3%. Duration: ≈ 3,600 seconds.
18:30 · Dependency outage
An outside service fails calls during an outage.
Failed calls: 100%. Duration: ≈ 1,200 seconds.
Play this problem. Keep one baseline, predict a change and compare the next full day. The in-game editorial has calibrated reference designs after you finish a traffic day.