Medium · AI, ads and search
Recommendations
Every page shows a row of things the visitor might like. A ranking model builds each row, but the model is slow and costly. The row must appear instantly, even when the model is having a bad day. Practise this medium system design problem for free with a Google or GitHub account.
The brief
Every page shows a row of things the visitor might like. A ranking model builds each row, but the model is slow and costly. The row must appear instantly, even when the model is having a bad day.
Think of a "you may also like" row.
Users, scale and requirements
- Who uses it
- 20 million people a month
- Audience in plain numbers
- 20,000,000 people per month
- Normal peak
- ≈ 15,000 requests per second
- Request mix
- Read: 93%; Analytics event: 7%
- Speed for 99 of 100 requests
- Read: ≈ 350 ms
- Successful requests
- 99.5%
- Data safety
- Best effort: small losses are tolerated.
- Monthly budget
- ≈ $19,000
- Data at rest
- ≈ 3 TB
- Where users are
- North America: 60%; Europe: 40%
- Home region
- North America
- Waiting deadlines
- Read: 900 seconds; Analytics event: 3,600 seconds
≈ marks an approximation, just as in the game. Waiting deadlines measure the age of queued work and are separate from interactive response time.
- A ranking call takes about 50 ms of model time (approx.).
- Click events help tomorrow's lists but need not be real-time.
Questions that shape the design
Does the row need to be exactly right for this second?
No. A list computed a few hours ago is fine for most people.
What if the model is slow or down?
Show a simple popular-items row. A generic row is far better than a blank space.
Is it a problem if two people see the same row?
No. Many people share a similar taste group, so shared answers are fine.
The traffic day
Consumer traffic, busiest in the evening. The clock covers 00:00 to 24:00 in the home region. Events add pressure on top of that normal traffic.
12:00 · Slow dependency
An outside service slows down. Your servers fill with requests waiting on it.
Added response time: ≈ 3,000 ms. Duration: ≈ 1,800 seconds.
20:00 · Cache stampede
The cache empties (restart, purge, mass expiry) and every request rushes to the database at once.
Recovery window: ≈ 300 seconds.
20:30 · Replication lag
Copies fall behind. Users read old data, even their own fresh changes.
Copy lag: ≈ 30 seconds. Duration: ≈ 1,200 seconds.
What a strong design has to handle
- Ranking is expensive. Which rows can be reused between visits?
- Bound ranking latency and use a compatible fallback when model or features fail.
- A longer cache time cuts model cost but shows older tastes. A new item takes a while to appear.
- The breaker trades quality for speed: a generic row is shown when the model is slow.
Make a prediction, run a full traffic day, then compare the result. The in-game editorial opens after a finished day and includes measured reference designs.
Concepts to practise
Related problems
- Hard
Ad serving
Each page view asks you to pick the best ad in about a quarter of a second: check who is allowed to see what, score the candidates with a model, run a quick auction and answer. Every impression and click must be counted exactly because money depends on it.
- Easy
Semantic search
A help center lets people type a question in plain words and finds the right articles even when the words differ. Each question is turned into a list of numbers (a vector) and matched to similar articles. Many people ask the same few questions.
- Hard
AI chatbot serving
Millions of people ask your assistant questions. Each reply takes a GPU several seconds, and GPUs are expensive and sometimes simply unavailable. Keep replies flowing, protect your wallet from abuse, and have a plan for when the GPUs run out.
- Easy
URL shortener
People paste a long web address and get back a short one. Millions of other people then tap that short link, so reading is almost everything. Keep every tap fast, and never lose a link.