Hard · Media and streaming
Video upload pipeline
Creators upload videos that must be converted into several qualities before anyone can watch. Conversion is slow, costs real money per video, and the converter sometimes fails on a broken file. Nothing a creator uploads can be lost. Practise this hard system design problem for free with a Google or GitHub account.
The brief
Creators upload videos that must be converted into several qualities before anyone can watch. Conversion is slow, costs real money per video, and the converter sometimes fails on a broken file. Nothing a creator uploads can be lost.
Think of uploads on a site like YouTube.
Users, scale and requirements
- Who uses it
- 8 million viewers a month, 15 thousand uploads a day
- Audience in plain numbers
- 8,000,000 people per month
- Normal peak
- ≈ 150 requests per second
- Request mix
- Read: 60%; Stream chunk: 30%; Analytics event: 7.8%; Write: 2%; Upload: 0.2%
- Speed for 99 of 100 requests
- Read: ≈ 450 ms; Stream chunk: ≈ 1,000 ms; Write: ≈ 500 ms; Upload: ≈ 5,000 ms
- Successful requests
- 99.9%
- Data safety
- No acknowledged write may be lost.
- Monthly budget
- ≈ $740,000
- Data at rest
- ≈ 200 TB
- Where users are
- North America: 50%; Europe: 30%; Asia Pacific: 20%
- Home region
- North America
- Waiting deadlines
- Read: 900 seconds; Stream chunk: 900 seconds; Analytics event: 3,600 seconds; Write: 3,600 seconds; Upload: 600 seconds
- Fresh reads
- People must see their own changes. At most 0.1% of reads may be stale in the model.
- Personal reads
- 30% of reads are private; a shared public copy cannot answer them.
≈ marks an approximation, just as in the game. Waiting deadlines measure the age of queued work and are separate from interactive response time.
- Converting one 10-minute video into five qualities costs about $1 at list price (approx.). Conversion is the largest line on the bill.
- One upload request in the game stands for one whole video handed over to be converted.
Questions that shape the design
Must the creator wait while the video converts?
No. The upload is accepted right away. The video shows as "processing" and goes live when ready, within about ten minutes.
What if the same upload is sent twice because the connection dropped?
It must not be converted twice. Each conversion costs about a dollar.
What if one file is corrupted?
It will fail every time. It must not block the other uploads behind it.
What if the converter service is down for a while?
Uploads still succeed and wait in line. They convert once it comes back.
The traffic day
Consumer traffic, busiest in the evening. The clock covers 00:00 to 24:00 in the home region. Events add pressure on top of that normal traffic.
18:00 · Viral spike
A post blows up. Traffic climbs to several times normal within minutes.
Traffic reaches 6 times the scheduled load; rises over 600 seconds, holds for 3,600 seconds and eases over 3,600 seconds.
10:00 · Poison message
One malformed message crashes every worker that picks it up, again and again.
Lasts until detected and repaired.
15:00 · Dependency outage
An outside service fails calls during an outage.
Failed calls: 100%. Duration: ≈ 2,400 seconds.
What a strong design has to handle
- The converter is slow, costly and sometimes broken. What goes between the upload and the converter?
- Make upload acceptance durable and conversion repeatable despite poison files.
- Accepting the upload before converting means a creator sees "processing" for minutes, but the upload never fails because the converter is slow.
- Retrying a failed conversion fixes temporary problems but wastes money on a broken file. A retry limit plus a dead-letter queue stops that.
Make a prediction, run a full traffic day, then compare the result. The in-game editorial opens after a finished day and includes measured reference designs.
Concepts to practise
Related problems
- Medium
Notification service
Every other team asks you to send messages: order shipped, new login, a sale starting. You send them by push, email and text through outside providers who have their own limits. Send each message once, and never block the team that asked.
- Hard
Scheduled job service
Customers register jobs: "call my web address every night at 02:00" or "every hour on the hour". Aim to fire within 5 seconds; work still waiting after 60 seconds counts as failed. Retry safely after a crash. One external effect requires the customer endpoint to honor a stable job occurrence key. Half the world picks the top of the hour, so a huge crowd of jobs comes due in the same second.
- Easy
Invoicing and billing
Small businesses create invoices, email them to customers and take card payments online. On the last working day of the month everyone sends and chases invoices at once. Nobody may get the same invoice email twice, and no invoice may vanish.
- Easy
Payroll run
On pay day your system tells the payment rail to move salaries for two million employees, then publishes a payslip for each one. Employees open the app at breakfast to see if they were paid. Paying anyone twice is the one thing you must never do.