Medium · Developer platforms
CI/CD build system
Every time a developer saves code to the shared repository, you build and test it. Thousands of developers do this all day, and release day brings a rush. Builds must start quickly, but idle build machines cost real money. Practise this medium system design problem for free with a Google or GitHub account.
The brief
Every time a developer saves code to the shared repository, you build and test it. Thousands of developers do this all day, and release day brings a rush. Builds must start quickly, but idle build machines cost real money.
Think of build runners like GitHub Actions.
Users, scale and requirements
- Who uses it
- 40,000 developers
- Audience in plain numbers
- 40,000 users or devices
- Normal peak
- ≈ 300 requests per second
- Request mix
- Analytics event: 50%; Batch task: 20%; Read: 20%; Write: 10%
- Speed for 99 of 100 requests
- Read: ≈ 300 ms; Write: ≈ 300 ms
- Successful requests
- 99.5%
- Data safety
- At most 1 minute of writes may be lost.
- Monthly budget
- ≈ $106,000
- Data at rest
- ≈ 120 TB
- Where users are
- North America: 60%; Europe: 40%
- Home region
- North America
- Waiting deadlines
- Analytics event: 3,600 seconds; Batch task: 120 seconds; Read: 900 seconds; Write: 3,600 seconds
- Minimum retained history
- 30 days on authoritative stores
≈ marks an approximation, just as in the game. Waiting deadlines measure the age of queued work and are separate from interactive response time.
- A build takes about 5 minutes and produces a 5 MB artifact. Convert peak batch arrivals into concurrent work.
- A "batch" request here is one build job.
- Scored data retention: at least 30 days on active authoritative stores. Copies used only for search or reports can be rebuilt. Local device history and rollup policies still require the separate protocol check.
Questions that shape the design
How long may a build wait to start?
Under a few minutes. Developers get impatient after about two.
What if one broken build keeps crashing the workers?
Stop retrying it after a few tries, set it aside and tell the author. It must not block anyone else.
Do we keep build outputs?
Yes, for a month, in cheap storage.
May we use spare cheap machines that can disappear?
Yes. A build can be restarted if its machine vanishes.
The traffic day
Traffic concentrated in working hours. The clock covers 00:00 to 24:00 in the home region. Events add pressure on top of that normal traffic.
10:00 · Flash crowd
A scheduled launch brings a sudden rush of requests.
Traffic reaches 6 times the scheduled load; rises over 60 seconds, holds for 1,800 seconds and eases over 1,800 seconds.
14:00 · Poison message
One malformed message crashes every worker that picks it up, again and again.
Lasts until detected and repaired.
17:00 · Cost blowout
A crawler hits uncached pages all day, and every pay-per-use part of the bill multiplies.
Traffic for Read, Search reaches 1.5 times the scheduled load; rises over 600 seconds, holds for 21,600 seconds and eases over 600 seconds. Other request types keep their scheduled load. Billed usage multiplier: 3. Duration: ≈ 21,600 seconds.
What a strong design has to handle
- Builds arrive in bursts but workers cost money by the minute. How do you smooth that out?
- Schedule isolated retryable builds and publish artifacts only from the correct commit.
- Keeping a few workers warm starts builds faster but costs money all night.
- Cheap interruptible machines save a lot but need builds that can restart safely.
Make a prediction, run a full traffic day, then compare the result. The in-game editorial opens after a finished day and includes measured reference designs.
Concepts to practise
Related problems
- Medium
Notification service
Every other team asks you to send messages: order shipped, new login, a sale starting. You send them by push, email and text through outside providers who have their own limits. Send each message once, and never block the team that asked.
- Medium
Patient health records
A hospital network keeps patient charts, orders, notes and scanned documents for 40 hospitals. Doctors need a chart in under a second, no saved note may be lost when a server or a building fails (in a whole-region disaster, at most the last minute of edits may need re-entering from the ward), and an auditor can walk in any day and ask you to prove who looked at what.
- Hard
Tax filing deadline
A national tax site lets 30 million people file online. Most wait until deadline day, and traffic climbs all evening to a final-hour rush. People upload forms, check what they owe and pay. A mistake in a payment or a leak of tax data is a national story.
- Hard
Email service
Mail arrives from the whole internet, including a lot of spam, and must land in the right inbox. People open, search, send and attach. Never lose a message, and keep search fast even over years of mail.