Medium · Developer platforms
Clickstream analytics
Every tap and page view on a busy site sends a tiny event to you. Hundreds of millions pile up each day. Analysts open dashboards to ask what happened this hour. Never drop an event, and keep the storage bill under control as the pile grows. Practise this medium system design problem for free with a Google or GitHub account.
The brief
Every tap and page view on a busy site sends a tiny event to you. Hundreds of millions pile up each day. Analysts open dashboards to ask what happened this hour. Never drop an event, and keep the storage bill under control as the pile grows.
Think of the page-view tracking behind a product analytics tool.
Users, scale and requirements
- Who uses it
- Events from 30 million visitors a month (approx.)
- Audience in plain numbers
- 30,000,000 people per month
- Normal peak
- ≈ 8,000 requests per second
- Request mix
- Analytics event: 99%; Report: 1%
- Speed for 99 of 100 requests
- Report: ≈ 30,000 ms
- Successful requests
- 99.9%
- Data safety
- No acknowledged write may be lost.
- Monthly budget
- ≈ $72,000
- Data at rest
- ≈ 20 TB
- Where users are
- North America: 50%; Europe: 30%; Asia Pacific: 20%
- Home region
- North America
- Waiting deadlines
- Analytics event: 3,600 seconds; Report: 3,600 seconds
≈ marks an approximation, just as in the game. Waiting deadlines measure the age of queued work and are separate from interactive response time.
- About 300 million events a day, each about 0.5 KB (approx.).
- Roughly 150 GB of new raw data a day (approx.).
- All numbers are rough approximations.
Questions that shape the design
Must a dashboard be real-time?
A few minutes behind is fine. Nobody needs the exact second.
Can an event be counted twice?
Rarely and a little is acceptable for page views, but lost events are not.
How long do we keep raw events?
Recent data stays handy for a month. Older data is rarely read and can go to cheap storage.
The traffic day
Consumer traffic, busiest in the evening. The clock covers 00:00 to 24:00 in the home region. Events add pressure on top of that normal traffic.
20:00 · Viral spike
A post blows up. Traffic climbs to several times normal within minutes.
Traffic reaches 2 times the scheduled load; rises over 600 seconds, holds for 3,600 seconds and eases over 3,600 seconds.
14:00 · Metrics explosion
A metric label creates many more distinct series, increasing the bill.
Billed usage multiplier: 20. Duration: ≈ 21,600 seconds.
16:00 · Cost blowout
A crawler hits uncached pages all day, and every pay-per-use part of the bill multiplies.
Traffic for Read, Search reaches 1.5 times the scheduled load; rises over 600 seconds, holds for 21,600 seconds and eases over 600 seconds. Other request types keep their scheduled load. Billed usage multiplier: 3. Duration: ≈ 21,600 seconds.
What a strong design has to handle
- Events arrive far faster than people read dashboards. What can sit between the two?
- Acknowledge durable events and aggregate retries and late arrivals correctly.
- Dashboards a few minutes behind are far cheaper than live ones, because summaries are built in batches.
- Keeping raw events lets analysts ask new questions later, but storage grows every day. Tiering keeps it affordable.
Make a prediction, run a full traffic day, then compare the result. The in-game editorial opens after a finished day and includes measured reference designs.
Concepts to practise
Related problems
- Medium
Fleet telemetry
A freight company has 150,000 trucks that each send a batch of sensor readings every 30 seconds. Dispatchers watch live maps and analysts run reports. You must take in every batch, keep a month of detail cheaply, and survive the moments when thousands of trucks come back online at once.
- Medium
Smart power grid
A power company has 5 million smart meters. Each reports usage every 15 minutes, and they all try on the quarter hour. Customers see their usage, billing needs exact totals for seven years, and a regulator checks that records are kept safely. Control commands to home batteries and thermostats must never run twice.
- Hard
Email service
Mail arrives from the whole internet, including a lot of spam, and must land in the right inbox. People open, search, send and attach. Never lose a message, and keep search fast even over years of mail.
- Medium
CI/CD build system
Every time a developer saves code to the shared repository, you build and test it. Thousands of developers do this all day, and release day brings a rush. Builds must start quickly, but idle build machines cost real money.