Hard · Core building blocks
Scheduled job service
Customers register jobs: "call my web address every night at 02:00" or "every hour on the hour". Aim to fire within 5 seconds; work still waiting after 60 seconds counts as failed. Retry safely after a crash. One external effect requires the customer endpoint to honor a stable job occurrence key. Half the world picks the top of the hour, so a huge crowd of jobs comes due in the same second. Practise this hard system design problem for free with a Google or GitHub account.
The brief
Customers register jobs: "call my web address every night at 02:00" or "every hour on the hour". Aim to fire within 5 seconds; work still waiting after 60 seconds counts as failed. Retry safely after a crash. One external effect requires the customer endpoint to honor a stable job occurrence key. Half the world picks the top of the hour, so a huge crowd of jobs comes due in the same second.
Think of a cron-as-a-service product.
Users, scale and requirements
- Who uses it
- 300,000 customers and 40 million jobs (approx.)
- Audience in plain numbers
- 300,000 users or devices
- Normal peak
- ≈ 1,200 requests per second
- Request mix
- Webhook: 70%; Write: 15%; Read: 15%
- Speed for 99 of 100 requests
- Write: ≈ 1,500 ms; Read: ≈ 1,500 ms
- Successful requests
- 99.9%
- Data safety
- No acknowledged write may be lost.
- Monthly budget
- ≈ $7,700
- Data at rest
- ≈ 0.3 TB
- Where users are
- North America: 60%; Europe: 40%
- Home region
- North America
- Waiting deadlines
- Webhook: 60 seconds; Write: 60 seconds; Read: 60 seconds
- Fresh reads
- People must see their own changes. At most 0.1% of reads may be stale in the model.
≈ marks an approximation, just as in the game. Waiting deadlines measure the age of queued work and are separate from interactive response time.
- A job record is about 1 KB (approx.).
- At the top of the hour about 1,200 requests a second come due for a few minutes (approx.).
- One app server handles about 1,000 requests a second (approx.).
- All numbers are rough approximations.
- Customer web addresses have an aggregate quota of 10,000 webhooks/s across independent endpoints, with no per-call infrastructure fee. One endpoint can still time out. External one-effect delivery requires endpoint idempotency or reconciliation.
Questions that shape the design
How late may a job fire?
The webhook p99 target is 5,000 ms. Work still waiting after 60 seconds counts as failed, even if it eventually runs.
What if a job is delivered twice?
That is the worst outcome: a bill sent twice, a backup run twice. Twice must never happen, even if a worker crashes and the job is tried again.
Do jobs we cannot run get lost?
No. A job that keeps failing is parked where the customer can see it, and everything else carries on.
The traffic day
Fairly steady traffic throughout the day. The clock covers 00:00 to 24:00 in the home region. Events add pressure on top of that normal traffic.
12:00 · Flash crowd
A scheduled launch brings a sudden rush of requests.
Traffic reaches 3 times the scheduled load; rises over 30 seconds, holds for 900 seconds and eases over 1,800 seconds.
15:00 · Duplicate delivery
The queue redelivers messages. Without duplicate protection, side effects may be applied twice.
Affected share: 1%. Duration: ≈ 3,600 seconds.
What a strong design has to handle
- Jobs come due in a wall at the top of the hour. What can hold the wall so the servers are not flooded?
- Discover due occurrences, fence job ownership, and park permanent failures.
- Firing a few seconds early and spreading the wall over a short window flattens the peak, at the price of jobs that are not to the second.
- A local lookup alone cannot guarantee one external effect after a crash. Use the same job occurrence key at a cooperating endpoint, or expose the unknown outcome for reconciliation.
Make a prediction, run a full traffic day, then compare the result. The in-game editorial opens after a finished day and includes measured reference designs.
Concepts to practise
Related problems
- Medium
Notification service
Every other team asks you to send messages: order shipped, new login, a sale starting. You send them by push, email and text through outside providers who have their own limits. Send each message once, and never block the team that asked.
- Easy
Lab results portal
Labs send results to a portal, patients get a notice that a result is ready and open it. When the notice goes to thousands of people at the same minute, they all open the portal at once. Nobody should get the same notice twice.
- Hard
Concert ticket drop
Tickets for a huge tour go on sale at 10:00 sharp. Hundreds of thousands of fans arrive in the same minute, bots arrive earlier, and everyone refreshes when the page stalls. There are 40,000 seats. Sell each one once, charge each buyer once, and keep the site standing.
- Hard
Video upload pipeline
Creators upload videos that must be converted into several qualities before anyone can watch. Conversion is slow, costs real money per video, and the converter sometimes fails on a broken file. Nothing a creator uploads can be lost.