{"groups":[{"id":"core","title":"Core building blocks","blurb":"The small services every product leans on: links, limits, search boxes and notifications.","problems":["url-shortener","rate-limiter-service","search-autocomplete","notification-service","login-sso","job-scheduler","object-storage"]},{"id":"media","title":"Media and streaming","blurb":"Photos, music, podcasts, news and video: big files, huge crowds and a bandwidth bill.","problems":["photo-sharing","podcast-hosting","news-site","music-streaming","video-on-demand","short-video-feed","video-upload-pipeline","live-streaming"]},{"id":"social","title":"Social and messaging","blurb":"Likes, chats, stories and feeds: tiny messages with enormous fanout.","problems":["comments-and-likes","chat-app","stories","social-feed","group-chat-presence"]},{"id":"fintech","title":"Finance and payments","blurb":"Money moves: every write must land exactly once and survive a failure.","problems":["invoicing-billing","payroll-run","payments-checkout","digital-wallet","insurance-claims","bnpl-credit","bank-ledger","stock-order-matching","fraud-detection","crypto-exchange"]},{"id":"health","title":"Healthcare","blurb":"Bookings, records and video visits, with privacy rules and sudden crowds.","problems":["appointment-booking","lab-results","health-records","telehealth-video","pharmacy-stock","vaccine-signup-surge"]},{"id":"civic","title":"Government and education","blurb":"Deadline days and exam mornings, when everyone arrives at the same minute.","problems":["benefits-portal","course-enrollment-day","online-exam","tax-deadline-surge"]},{"id":"mobility","title":"Mobility and logistics","blurb":"Drivers, trucks and parcels sending their position every few seconds.","problems":["package-tracking","fleet-telemetry","food-delivery","ride-hailing-dispatch","maps-routing"]},{"id":"travel","title":"Travel and events","blurb":"Seats, rooms and tickets: a few hot items and many people trying for them.","problems":["restaurant-reservations","hotel-booking","airline-booking","concert-ticket-drop"]},{"id":"iot","title":"IoT and devices","blurb":"Millions of small devices that wake up together.","problems":["ev-charging-network","smart-home-hub","smart-power-grid","industrial-sensors","firmware-rollout"]},{"id":"ai","title":"AI, ads and search","blurb":"Model serving, ranking and auctions inside a tight time budget.","problems":["semantic-search","recommendations","ad-serving","ai-chatbot-serving"]},{"id":"games","title":"Games","blurb":"Leaderboards and matchmaking where every player wants an answer now.","problems":["leaderboards","matchmaking","game-patch-day","battle-royale"]},{"id":"productivity","title":"Productivity","blurb":"Files, documents and mail that people trust to never lose a word.","problems":["calendar-invites","file-sync","collaborative-docs","email-service"]},{"id":"commerce","title":"Commerce and retail","blurb":"Shopping carts and sale days: lots of browsing, few buys, and every unit of stock counted once.","problems":["shopping-cart","flash-sale-inventory"]},{"id":"telecom","title":"Telecom and messaging","blurb":"Texts and calls: carriers with hard limits, and codes that must arrive in seconds.","problems":["otp-sms-delivery","call-routing"]},{"id":"devtools","title":"Developer platforms","blurb":"Builds, flags and metrics: the tools that run the other tools.","problems":["status-page","ci-cd-pipeline","feature-flag-service","clickstream-analytics","observability-platform"]}],"problems":[{"id":"url-shortener","title":"URL shortener","difficulty":"Easy","concepts":["caching","back-of-envelope"],"like":"a link shortener like Bitly","brief":"People paste a long web address and get back a short one. Millions of other people then tap that short link, so reading is almost everything. Keep every tap fast, and never lose a link.","group":"core","daily":true,"requires":{},"palette":null,"events":[{"id":"viral-spike","at":"19:30","params":{"factor":3}},{"id":"cache-stampede","at":"12:10"}],"eventPool":["hot-key","db-failover"]},{"id":"rate-limiter-service","title":"Rate limiter for a public API","difficulty":"Medium","concepts":["rate-limiting","hot-keys"],"like":"the limits on a public API like GitHub's","brief":"Developers call your public API from scripts and apps. A few misbehaving or malicious clients can crowd out everyone else. Build limits that are fair, fast and cheap, and that survive a retry storm.","group":"core","daily":true,"requires":{"read":[["rate-limiter","api-gateway"]],"write":[["rate-limiter","api-gateway"]]},"palette":null,"events":[{"id":"bot-scraping","at":"10:00","params":{"factorOfPeak":1,"durationSec":7200}},{"id":"retry-storm","at":"15:30"}],"eventPool":["hot-key","ddos-attack"]},{"id":"search-autocomplete","title":"Search autocomplete","difficulty":"Medium","concepts":["caching","search-indexing"],"like":"suggestions as you type, like Google","brief":"As people type, they expect suggestions after every letter, in the blink of an eye. Most people type the same popular beginnings. When news breaks, one phrase suddenly dominates.","group":"core","daily":true,"requires":{},"palette":null,"events":[{"id":"viral-spike","at":"14:00","params":{"factor":6}},{"id":"hot-key","at":"14:05","params":{"share":0.4}}],"eventPool":["cache-stampede","bad-deploy"]},{"id":"notification-service","title":"Notification service","difficulty":"Medium","concepts":["queues-backpressure","idempotency-retries"],"like":"the service that sends every push, email and text","brief":"Every other team asks you to send messages: order shipped, new login, a sale starting. You send them by push, email and text through outside providers who have their own limits. Send each message once, and never block the team that asked.","group":"core","daily":true,"requires":{},"palette":null,"events":[{"id":"third-party-rate-limit","at":"09:00","params":{"rpsCapFactor":0.4}},{"id":"duplicate-delivery","at":"13:00","params":{"share":0.03}},{"id":"dependency-outage","at":"18:30"}],"eventPool":["flash-crowd","poison-message"]},{"id":"login-sso","title":"Login and single sign-on","difficulty":"Medium","concepts":["rate-limiting","security-compliance"],"like":"the \"Sign in with\" button used by many apps","brief":"Employees and customers sign in to many apps through one account. Most of the day is quiet, then 9 am arrives and everyone signs in at once. Attackers also hammer the login with stolen passwords. Let real people in quickly and keep the rest out.","group":"core","daily":true,"requires":{},"palette":null,"events":[{"id":"bot-scraping","at":"10:30"},{"id":"retry-storm","at":"09:00"}],"eventPool":["cert-expiry","ddos-attack","key-leak"]},{"id":"job-scheduler","title":"Scheduled job service","difficulty":"Hard","concepts":["queues-backpressure","idempotency-retries"],"like":"a cron-as-a-service product","brief":"Customers register jobs: \"call my web address every night at 02:00\" or \"every hour on the hour\". Your service must fire each job on time, exactly once, even when a machine dies halfway. Half the world picks the top of the hour, so a huge crowd of jobs comes due in the same second.","group":"core","daily":false,"requires":{},"palette":null,"events":[{"id":"flash-crowd","at":"12:00","params":{"factor":3}},{"id":"duplicate-delivery","at":"15:00"}],"eventPool":["poison-message","db-failover"]},{"id":"object-storage","title":"Object storage service","difficulty":"Hard","concepts":["storage-tiers","sharding"],"like":"a cloud file store like Amazon S3","brief":"Customers store billions of files, from tiny pictures to huge videos, and fetch them by name. The service must never lose a file, scale without a limit and stay cheap. Most files are written once and rarely read again.","group":"core","daily":true,"requires":{},"palette":null,"events":[{"id":"viral-spike","at":"10:00","params":{"factor":3}},{"id":"zone-outage","at":"15:00"},{"id":"cost-blowout","at":"19:00"}],"eventPool":["hot-key","slow-dependency"]},{"id":"photo-sharing","title":"Photo sharing","difficulty":"Easy","concepts":["cdn","storage-tiers"],"like":"a photo app like Instagram","brief":"You are building a photo app. People post pictures and everyone else scrolls through them. Almost all the traffic is people looking at photos, and the photos take up far more room than anything else you store.","group":"media","daily":true,"requires":{},"palette":null,"events":[{"id":"viral-spike","at":"20:00","params":{"factor":6}},{"id":"cache-stampede","at":"14:00"}],"eventPool":["cost-blowout","bot-scraping"]},{"id":"podcast-hosting","title":"Podcast hosting","difficulty":"Easy","concepts":["cdn","cost"],"like":"a podcast host","brief":"You host shows for creators. Listeners press play and download big audio files, and creators upload a new episode now and then. The audio is the bill: every episode downloaded costs bandwidth.","group":"media","daily":true,"requires":{},"palette":null,"events":[{"id":"flash-crowd","at":"07:30","params":{"factor":10}},{"id":"cost-blowout","at":"12:00"}],"eventPool":["viral-spike","bot-scraping"]},{"id":"news-site","title":"News site","difficulty":"Easy","concepts":["caching","cdn"],"like":"a news site like the BBC or CNN","brief":"You run a news site. Millions read the same few stories, and when something big happens almost everyone opens the same page in the same minute. Editors publish all day, and readers expect fresh headlines.","group":"media","daily":true,"requires":{},"palette":null,"events":[{"id":"flash-crowd","at":"13:00","params":{"factor":25}},{"id":"cache-stampede","at":"09:00"}],"eventPool":["bot-scraping","ddos-attack"]},{"id":"music-streaming","title":"Music streaming","difficulty":"Medium","concepts":["cdn","caching","search-indexing"],"like":"a music service like Spotify","brief":"People search for songs, build playlists and press play. The audio is the heavy part, and a hit song is played by millions at once. Playback must not stutter, even when part of your system fails.","group":"media","daily":true,"requires":{},"palette":null,"events":[{"id":"hot-key","at":"10:00","params":{"share":0.35}},{"id":"region-outage","at":"15:00","params":{"durationSec":1800}}],"eventPool":["viral-spike","cache-stampede","slow-dependency"]},{"id":"video-on-demand","title":"Video streaming","difficulty":"Medium","concepts":["cdn","cost","storage-tiers"],"like":"a video service like Netflix","brief":"You run a movie and series service for a growing audience. Each viewer pulls a few seconds of video at a time for two hours straight. Bandwidth is the largest cost by far, and everyone wants to start watching in the evening.","group":"media","daily":true,"requires":{},"palette":null,"events":[{"id":"flash-crowd","at":"19:00","params":{"factor":8}},{"id":"cache-stampede","at":"12:00"},{"id":"cost-blowout","at":"21:00"}],"eventPool":["region-outage","viral-spike","dependency-outage"]},{"id":"short-video-feed","title":"Short-video feed","difficulty":"Medium","concepts":["cdn","caching","hot-keys"],"like":"a short-video app like TikTok","brief":"People swipe through an endless feed of clips of 10 to 30 seconds. The next clip must already be on the phone before the swipe. One clip can suddenly get millions of views, and every view is a signal that picks what to show next.","group":"media","daily":true,"requires":{},"palette":null,"events":[{"id":"viral-spike","at":"20:15","params":{"factor":5}},{"id":"hot-key","at":"20:20","params":{"share":0.4}},{"id":"slow-dependency","at":"11:00"}],"eventPool":["cache-stampede","cost-blowout","dependency-outage"]},{"id":"video-upload-pipeline","title":"Video upload pipeline","difficulty":"Hard","concepts":["queues-backpressure","idempotency-retries"],"like":"uploads on a site like YouTube","brief":"Creators upload videos that must be converted into several qualities before anyone can watch. Conversion is slow, costs real money per video, and the converter sometimes fails on a broken file. Nothing a creator uploads can be lost.","group":"media","daily":true,"requires":{},"palette":null,"events":[{"id":"viral-spike","at":"18:00","params":{"factor":6}},{"id":"poison-message","at":"10:00"},{"id":"dependency-outage","at":"15:00","params":{"durationSec":2400}}],"eventPool":["duplicate-delivery","retry-storm","disk-full"]},{"id":"live-streaming","title":"Live streaming","difficulty":"Hard","concepts":["cdn","realtime-connections","fanout"],"like":"a live platform like Twitch","brief":"Streamers broadcast live and thousands watch together, with a chat on the side. Everyone wants the same piece of video at the same second, so nothing can be saved for later. When a big streamer goes live, the whole audience arrives at once.","group":"media","daily":true,"requires":{},"palette":null,"events":[{"id":"thundering-herd","at":"20:00"},{"id":"flash-crowd","at":"20:30","params":{"factor":8}},{"id":"region-outage","at":"21:30","params":{"durationSec":1200}}],"eventPool":["network-partition","ddos-attack","cache-stampede"]},{"id":"comments-and-likes","title":"Comments and likes","difficulty":"Easy","concepts":["hot-keys","caching"],"like":"the like counter under a post","brief":"Under every post sits a like counter and a comment list. Millions tap the heart on the same famous post at the same moment. Showing the exact number is not essential, but showing the person their own like is.","group":"social","daily":true,"requires":{},"palette":null,"events":[{"id":"viral-spike","at":"20:00","params":{"factor":5}},{"id":"hot-key","at":"13:00","params":{"share":0.1}}],"eventPool":["cache-stampede","replication-lag","bot-scraping"]},{"id":"chat-app","title":"One-to-one chat","difficulty":"Medium","concepts":["realtime-connections","idempotency-retries"],"like":"a chat app like WhatsApp","brief":"People send short messages to each other from their phones. A message must arrive once, in order, even when the phone is on a bad connection and sends it again. If the other person is offline, the message waits and a notification goes out.","group":"social","daily":true,"requires":{},"palette":null,"events":[{"id":"thundering-herd","at":"08:00"},{"id":"zone-outage","at":"14:00"},{"id":"duplicate-delivery","at":"19:00"}],"eventPool":["third-party-rate-limit","retry-storm","network-partition"]},{"id":"stories","title":"Disappearing stories","difficulty":"Medium","concepts":["storage-tiers","cdn"],"like":"24-hour stories","brief":"People post photos and short clips that vanish after 24 hours. Followers tap through a tray of stories every day, and the poster sees who looked. Deleting on time matters: storage must not grow forever.","group":"social","daily":true,"requires":{},"palette":null,"events":[{"id":"viral-spike","at":"19:00","params":{"factor":5}},{"id":"cost-blowout","at":"13:00"}],"eventPool":["cache-stampede","hot-key","bad-deploy"]},{"id":"social-feed","title":"Social feed","difficulty":"Hard","concepts":["fanout","caching","hot-keys"],"like":"a home timeline like X or Facebook","brief":"Every person opens the app and sees a feed of posts from the people they follow. Most people have a few hundred followers, but a few have millions, and when they post, millions of feeds must change. The feed must be fast and your own post must show up at once.","group":"social","daily":true,"requires":{},"palette":null,"events":[{"id":"celebrity-fanout","at":"12:00"},{"id":"hot-key","at":"17:30","params":{"share":0.3}},{"id":"cache-stampede","at":"08:15"}],"eventPool":["replication-lag","viral-spike","bad-deploy"]},{"id":"group-chat-presence","title":"Group chat and presence","difficulty":"Hard","concepts":["realtime-connections","fanout","consistency"],"like":"team chat like Slack or Discord","brief":"Teams talk in channels with many people. Everyone sees messages in the same order, sees who is online and who is typing, and gets notified when mentioned. At nine every morning thousands of workplaces start at once, and one announcement can reach a channel of tens of thousands.","group":"social","daily":true,"requires":{},"palette":null,"events":[{"id":"thundering-herd","at":"09:00"},{"id":"celebrity-fanout","at":"14:00","params":{"factor":20}},{"id":"network-partition","at":"16:00"}],"eventPool":["compliance-audit","replication-lag","zone-outage"]},{"id":"invoicing-billing","title":"Invoicing and billing","difficulty":"Easy","concepts":["queues-backpressure","idempotency-retries"],"like":"an invoicing tool like QuickBooks","brief":"Small businesses create invoices, email them to customers and take card payments online. On the last working day of the month everyone sends and chases invoices at once. Nobody may get the same invoice email twice, and no invoice may vanish.","group":"fintech","daily":true,"requires":{"payment":[["payment-processor"]]},"palette":null,"events":[{"id":"duplicate-delivery","at":"09:00","params":{}},{"id":"deadline-surge","at":"10:00","params":{"factor":3}},{"id":"third-party-rate-limit","at":"15:00","params":{}}],"eventPool":["cache-stampede","dependency-outage","viral-spike"]},{"id":"payroll-run","title":"Payroll run","difficulty":"Easy","concepts":["idempotency-retries","queues-backpressure"],"like":"a payroll service like Gusto","brief":"On pay day your system tells the payment rail to move salaries for two million employees, then publishes a payslip for each one. Employees open the app at breakfast to see if they were paid. Paying anyone twice is the one thing you must never do.","group":"fintech","daily":true,"requires":{"payment":[["payment-processor"]]},"palette":null,"events":[{"id":"deadline-surge","at":"02:00","params":{"factor":4,"types":["read","login"]}},{"id":"third-party-rate-limit","at":"10:00","params":{}},{"id":"duplicate-delivery","at":"11:30","params":{}}],"eventPool":["dependency-outage","retry-storm"]},{"id":"payments-checkout","title":"Checkout payments","difficulty":"Medium","concepts":["idempotency-retries","circuit-breakers"],"like":"a checkout like Stripe Checkout","brief":"Online shops send you their customers at the moment they pay. You call a card processor, record the order and answer in a couple of seconds. When the processor is slow or down, shoppers refresh and tap Pay again.","group":"fintech","daily":true,"requires":{"payment":[["payment-processor"]]},"palette":null,"events":[{"id":"duplicate-delivery","at":"10:30","params":{}},{"id":"slow-dependency","at":"13:00","params":{}},{"id":"dependency-outage","at":"19:30","params":{}}],"eventPool":["viral-spike","retry-storm","compliance-audit"]},{"id":"digital-wallet","title":"Peer-to-peer wallet","difficulty":"Medium","concepts":["consistency","idempotency-retries","security-compliance"],"like":"a wallet app like Venmo","brief":"People split dinner by sending each other money from their phones. A transfer takes money from one balance and adds it to another, and the sender must see the new balance right away. A flaky mobile connection often makes the app send the same transfer twice.","group":"fintech","daily":true,"requires":{"payment":[["ledger-db","sharded-sql","distributed-sql","sql-primary"]]},"palette":null,"events":[{"id":"duplicate-delivery","at":"12:00","params":{}},{"id":"compliance-audit","at":"16:00","params":{"checks":["audit-log","encryption","access-control"]}},{"id":"db-failover","at":"20:30","params":{"target":"busiest:kind:store"}}],"eventPool":["viral-spike","replication-lag","retry-storm"]},{"id":"insurance-claims","title":"Insurance claims","difficulty":"Medium","concepts":["queues-backpressure","security-compliance"],"like":"filing a car insurance claim online","brief":"After a hailstorm, thousands of drivers file claims online, each with a handful of photos. The storm offer ends at 5 pm, so everyone files at the last minute. Every claim must be saved safely, and every customer gets a confirmation email.","group":"fintech","daily":true,"requires":{"upload":[["object-storage"]]},"palette":null,"events":[{"id":"deadline-surge","at":"10:00","params":{"factor":6}},{"id":"dependency-outage","at":"14:00","params":{}},{"id":"compliance-audit","at":"17:30","params":{"checks":["audit-log","encryption","pii-vault","retention"]}}],"eventPool":["duplicate-delivery","disk-full","poison-message"]},{"id":"bnpl-credit","title":"Buy now, pay later","difficulty":"Medium","concepts":["circuit-breakers","queues-backpressure","idempotency-retries"],"like":"a pay-in-installments service like Klarna","brief":"At checkout a shopper picks \"pay in four\". You score the shopper in a few seconds, then charge the first quarter on their card. The card network sometimes asks you to slow down, and when apps retry, the retries make it worse.","group":"fintech","daily":true,"requires":{"payment":[["payment-processor"],["model-server"]]},"palette":null,"events":[{"id":"third-party-rate-limit","at":"12:00","params":{}},{"id":"compliance-audit","at":"16:00","params":{"checks":["audit-log","encryption","consent"]}},{"id":"retry-storm","at":"19:00","params":{}}],"eventPool":["viral-spike","dependency-outage","duplicate-delivery"]},{"id":"bank-ledger","title":"Bank ledger","difficulty":"Hard","concepts":["consistency","security-compliance","multi-region"],"like":"the core ledger of a bank","brief":"You run the system of record for a regional bank: every deposit, withdrawal and card settlement is a posting that must be exact and permanent. Customers in two continents read balances all day, and regulators can ask for proof at any time. A bad day for you is a loss of even one posting.","group":"fintech","daily":true,"requires":{"payment":[["ledger-db","distributed-sql"]]},"palette":null,"events":[{"id":"db-failover","at":"10:15","params":{"target":"busiest:kind:store"}},{"id":"data-corruption","at":"13:00","params":{"target":"busiest:kind:store"}},{"id":"region-outage","at":"15:30","params":{"durationSec":3600}},{"id":"compliance-audit","at":"18:00","params":{"checks":["audit-log","encryption","backups","access-control","retention"]}}],"eventPool":["replication-lag","schema-migration-lock","key-leak"]},{"id":"stock-order-matching","title":"Stock order matching","difficulty":"Hard","concepts":["consistency","queues-backpressure"],"like":"an exchange like Nasdaq","brief":"Brokers and traders send buy and sell orders, and you pair them fairly: the earliest order at the best price goes first. At the market open a rush of orders arrives in seconds. Every accepted order must be in one agreed sequence, and no trade may be lost or invented.","group":"fintech","daily":true,"requires":{"write":[["event-stream"]]},"palette":null,"events":[{"id":"flash-crowd","at":"09:30","params":{"factor":3,"types":["write","read","login"],"rampSec":60,"holdSec":900,"decaySec":900}},{"id":"zone-outage","at":"11:00","params":{}},{"id":"network-partition","at":"13:30","params":{}}],"eventPool":["hot-key","retry-storm","bad-deploy"]},{"id":"fraud-detection","title":"Real-time fraud detection","difficulty":"Hard","concepts":["ai-serving","circuit-breakers"],"like":"card fraud checks at a bank","brief":"Every card payment asks your system \"is this fraud?\" and wants the answer within about 150 milliseconds. Each check looks up the cardholder's recent habits and runs a model. The holiday shopping rush doubles the volume, and a bad model update can start declining honest customers.","group":"fintech","daily":true,"requires":{"payment":[["model-server"]]},"palette":null,"events":[{"id":"bad-deploy","at":"10:00","params":{"target":"el:model-server"}},{"id":"slow-dependency","at":"14:00","params":{"target":"el:feature-store","addLatencyMs":2000,"durationSec":1800}},{"id":"viral-spike","at":"19:00","params":{"factor":2}}],"eventPool":["cold-start","gpu-shortage","retry-storm"]},{"id":"crypto-exchange","title":"Crypto exchange","difficulty":"Hard","concepts":["rate-limiting","consistency","security-compliance"],"like":"a crypto exchange like Coinbase","brief":"People trade coins, check prices and withdraw funds around the clock from every continent. A sudden price crash makes everyone log in and trade at once, attackers flood you hoping to knock the site over, and a leaked key could empty a wallet. Balances must never be wrong.","group":"fintech","daily":true,"requires":{"payment":[["ledger-db"]]},"palette":null,"events":[{"id":"ddos-attack","at":"13:00","params":{"factorOfPeak":15}},{"id":"flash-crowd","at":"16:00","params":{"factor":4,"types":["read","write","login"],"rampSec":30,"holdSec":900,"decaySec":1800}},{"id":"key-leak","at":"21:00","params":{}}],"eventPool":["bot-scraping","hot-key","region-outage"]},{"id":"appointment-booking","title":"Clinic appointment booking","difficulty":"Easy","concepts":["consistency","queues-backpressure"],"like":"booking a doctor's visit online","brief":"A group of 60 clinics lets patients find a free slot, book it and get a text reminder. Slots open at 8 am, and the phone lines used to melt. Two people must never get the same slot, and reminders must go out even when the text-message company has a bad hour.","group":"health","daily":true,"requires":{},"palette":null,"events":[{"id":"flash-crowd","at":"08:00","params":{"factor":6,"rampSec":60,"holdSec":600,"decaySec":900}},{"id":"third-party-rate-limit","at":"10:30","params":{"rpsCapFactor":0.1,"durationSec":2400}}],"eventPool":["slow-dependency","db-failover"]},{"id":"lab-results","title":"Lab results portal","difficulty":"Easy","concepts":["caching","queues-backpressure","idempotency-retries"],"like":"a patient portal that shows blood test results","brief":"Labs send results to a portal, patients get a notice that a result is ready and open it. When the notice goes to thousands of people at the same minute, they all open the portal at once. Nobody should get the same notice twice.","group":"health","daily":true,"requires":{},"palette":null,"events":[{"id":"duplicate-delivery","at":"03:00","params":{"share":0.01,"durationSec":3600}},{"id":"flash-crowd","at":"07:00","params":{"factor":10,"rampSec":60,"holdSec":600,"decaySec":1200}}],"eventPool":["slow-dependency","db-failover"]},{"id":"health-records","title":"Patient health records","difficulty":"Medium","concepts":["security-compliance","multi-region"],"like":"a hospital's electronic records","brief":"A hospital network keeps patient charts, orders, notes and scanned documents for 40 hospitals. Doctors need a chart in under a second, no saved note may be lost when a server or a building fails (in a whole-region disaster, at most the last minute of edits may need re-entering from the ward), and an auditor can walk in any day and ask you to prove who looked at what.","group":"health","daily":true,"requires":{},"palette":null,"events":[{"id":"key-leak","at":"09:30","params":{}},{"id":"compliance-audit","at":"11:00","params":{}},{"id":"region-outage","at":"14:00","params":{"durationSec":5400}}],"eventPool":["noisy-neighbor","db-failover","disk-full"]},{"id":"telehealth-video","title":"Telehealth video visits","difficulty":"Medium","concepts":["realtime-connections","multi-region"],"like":"a video doctor visit","brief":"Patients join a video visit with a doctor from their phone. Most calls go straight between the two people, but the platform runs the waiting room, chat, sign-in, the join link and a relay for the calls that cannot connect directly. A bad minute here means a patient staring at a spinner.","group":"health","daily":true,"requires":{},"palette":null,"events":[{"id":"thundering-herd","at":"09:00","params":{"factor":3,"holdSec":90}},{"id":"zone-outage","at":"10:15","params":{"durationSec":1800}},{"id":"dependency-outage","at":"14:00","params":{"durationSec":1200}}],"eventPool":["slow-dependency","viral-spike","cert-expiry"]},{"id":"pharmacy-stock","title":"Pharmacy stock and refills","difficulty":"Medium","concepts":["replication-lag","search-indexing"],"like":"a pharmacy chain's stock and refill system","brief":"A pharmacy chain with 4,000 stores tracks how many boxes of each medicine every store has, takes refill requests, and shows patients which nearby store has their medicine today. If the system says a store has a box that is gone, a patient makes a wasted trip.","group":"health","daily":true,"requires":{},"palette":null,"events":[{"id":"noisy-neighbor","at":"07:30","params":{"factor":0.4,"durationSec":2400}},{"id":"replication-lag","at":"11:00","params":{"lagSec":45,"durationSec":1200}},{"id":"hot-key","at":"17:00","params":{"share":0.35,"durationSec":3600}}],"eventPool":["db-failover","compliance-audit","viral-spike"]},{"id":"vaccine-signup-surge","title":"Vaccine sign-up surge","difficulty":"Hard","concepts":["rate-limiting","queues-backpressure"],"like":"a county vaccine sign-up site","brief":"A county opens 80,000 vaccine appointments at 8 am. Within seconds, thousands of people reload and bots try to grab slots. The site must stay up, never sell the same slot twice, and be fair to people on a phone with a poor signal.","group":"health","daily":true,"requires":{},"palette":null,"events":[{"id":"flash-crowd","at":"08:00","params":{"factor":40,"rampSec":20,"holdSec":900,"decaySec":1800}},{"id":"bot-scraping","at":"08:05","params":{"factorOfPeak":0.5,"types":["login","read","write"],"durationSec":3600}},{"id":"db-failover","at":"08:20","params":{"durationSec":90}}],"eventPool":["ddos-attack","third-party-rate-limit","bad-deploy"]},{"id":"benefits-portal","title":"Benefits application portal","difficulty":"Easy","concepts":["queues-backpressure","security-compliance"],"like":"a state benefits site","brief":"People apply for food, health or housing help on a state site. Forms are long, people save drafts and upload documents, and a caseworker team reviews each one. The last week of the month is busy, and an auditor checks the privacy and access controls once a year.","group":"civic","daily":true,"requires":{},"palette":null,"events":[{"id":"compliance-audit","at":"10:00","params":{"checks":["access-control","backups","audit-log","encryption","retention"]}},{"id":"deadline-surge","at":"12:00","params":{"factor":4,"rampSec":21600,"holdSec":3600,"decaySec":900}}],"eventPool":["slow-dependency","db-failover","disk-full"]},{"id":"course-enrollment-day","title":"Course enrollment day","difficulty":"Medium","concepts":["hot-keys","rate-limiting"],"like":"a university's registration day","brief":"Registration opens at 9 am for 40,000 students, and a handful of popular courses have only 30 seats each. Everyone clicks \"enroll\" at the same moment, and when the page is slow they click again. A seat must go to exactly one student.","group":"civic","daily":true,"requires":{},"palette":null,"events":[{"id":"flash-crowd","at":"09:00","params":{"factor":12,"rampSec":30,"holdSec":900,"decaySec":1800}},{"id":"hot-key","at":"09:00","params":{"share":0.3,"durationSec":1800}},{"id":"retry-storm","at":"09:05","params":{}}],"eventPool":["db-failover","slow-dependency","bad-deploy"]},{"id":"online-exam","title":"Online exam","difficulty":"Medium","concepts":["realtime-connections","multi-region"],"like":"a timed online exam for 200,000 students","brief":"A national board runs a two-hour online exam for 200,000 students in three regions at the same time. Answers save automatically every half minute, a live heartbeat shows the student is still there, and proctoring marks suspicious events. Every saved answer matters. A minute of outage at the wrong time is a national story.","group":"civic","daily":true,"requires":{},"palette":null,"events":[{"id":"thundering-herd","at":"08:55","params":{"factor":3,"holdSec":300,"types":["login","message","read"]}},{"id":"cert-expiry","at":"09:40","params":{}},{"id":"zone-outage","at":"10:10","params":{"durationSec":1800}}],"eventPool":["network-partition","bad-config-push","dns-misconfig"]},{"id":"tax-deadline-surge","title":"Tax filing deadline","difficulty":"Hard","concepts":["queues-backpressure","autoscaling","security-compliance"],"like":"a national tax filing site","brief":"A national tax site lets 30 million people file online. Most wait until deadline day, and traffic climbs all evening to a final-hour rush. People upload forms, check what they owe and pay. A mistake in a payment or a leak of tax data is a national story.","group":"civic","daily":true,"requires":{"payment":[["payment-processor"]]},"palette":null,"events":[{"id":"compliance-audit","at":"15:00","params":{"checks":["audit-log","encryption","pii-vault","backups","access-control","retention"]}},{"id":"deadline-surge","at":"18:00","params":{"factor":3,"rampSec":14400,"holdSec":3600,"decaySec":600}},{"id":"third-party-rate-limit","at":"22:00","params":{"rpsCapFactor":0.5,"durationSec":3600}}],"eventPool":["db-failover","bot-scraping","bad-deploy"]},{"id":"package-tracking","title":"Package tracking","difficulty":"Easy","concepts":["caching","queues-backpressure"],"like":"parcel tracking like UPS","brief":"You run the tracking page for a parcel company. People refresh it again and again while they wait for a delivery, and depots send a scan every time a parcel moves. Keep the page fast, lose at most the last minute of scans in a failure (scanners keep a copy and can resend), and make sure a broken text-message service cannot slow anything down.","group":"mobility","daily":true,"requires":{},"palette":null,"events":[{"id":"dependency-outage","at":"10:15"},{"id":"viral-spike","at":"18:30","params":{"factor":4}}],"eventPool":["cache-stampede","duplicate-delivery","slow-dependency"]},{"id":"fleet-telemetry","title":"Fleet telemetry","difficulty":"Medium","concepts":["queues-backpressure","storage-tiers","observability"],"like":"trucks reporting every few seconds","brief":"A freight company has 150,000 trucks that each send a batch of sensor readings every 30 seconds. Dispatchers watch live maps and analysts run reports. You must take in every batch, keep a month of detail cheaply, and survive the moments when thousands of trucks come back online at once.","group":"mobility","daily":true,"requires":{},"palette":null,"events":[{"id":"thundering-herd","at":"07:00","params":{"factor":4,"types":["telemetry"]}},{"id":"cardinality-explosion","at":"11:00"},{"id":"disk-full","at":"14:30"}],"eventPool":["network-partition","bad-config-push","noisy-neighbor"]},{"id":"food-delivery","title":"Food delivery","difficulty":"Medium","concepts":["realtime-connections","idempotency-retries","circuit-breakers"],"like":"an app like DoorDash","brief":"You run food delivery in a few big cities. People browse menus, pay for an order, and watch a courier move on a map. Dinner time is brutal, and the payment company sometimes slows down. Nobody may be charged twice, and no paid order may vanish.","group":"mobility","daily":true,"requires":{"payment":[["payment-processor"]]},"palette":null,"events":[{"id":"viral-spike","at":"12:00","params":{"factor":2.5}},{"id":"slow-dependency","at":"18:30"},{"id":"duplicate-delivery","at":"19:15"}],"eventPool":["dependency-outage","zone-outage","cache-stampede"]},{"id":"ride-hailing-dispatch","title":"Ride-hailing dispatch","difficulty":"Hard","concepts":["realtime-connections","hot-keys","sharding"],"like":"an app like Uber","brief":"You run dispatch for a big metro area. About 100,000 drivers send a position every 4 seconds, and riders ask for cars near them. When a stadium empties, thousands of riders ask for a car from the same few blocks at the same moment. Match riders to drivers fast, and never charge a trip twice.","group":"mobility","daily":true,"requires":{"payment":[["payment-processor"]]},"palette":null,"events":[{"id":"zone-outage","at":"08:10"},{"id":"hot-key","at":"17:45","params":{"share":0.3,"durationSec":3600}},{"id":"flash-crowd","at":"22:45","params":{"factor":6,"rampSec":60,"holdSec":1800,"decaySec":1800,"types":["read","write","notify"]}}],"eventPool":["slow-dependency","retry-storm","replication-lag"]},{"id":"maps-routing","title":"Maps and routing","difficulty":"Hard","concepts":["cdn","caching","multi-region"],"like":"a maps app like Google Maps","brief":"You run a maps app with places, routes and live traffic. Map pictures never change much, but route lookups, place search and traffic from moving phones do. Soon a whole new continent will start using it. Keep maps fast everywhere without paying to serve the same picture twice.","group":"mobility","daily":true,"requires":{},"palette":null,"events":[{"id":"overseas-launch","at":"08:00"},{"id":"viral-spike","at":"13:00","params":{"factor":4,"types":["read","search"]}},{"id":"cache-stampede","at":"17:00"}],"eventPool":["hot-key","region-outage","slow-dependency"]},{"id":"restaurant-reservations","title":"Restaurant reservations","difficulty":"Easy","concepts":["consistency","caching"],"like":"a booking site like OpenTable","brief":"Diners look up restaurants, check which tables are free tonight and book one. Looking is easy to speed up. Booking is not: two people must never get the same table. Friday at 5 pm is the busy moment.","group":"travel","daily":true,"requires":{},"palette":null,"events":[{"id":"flash-crowd","at":"17:00","params":{"factor":3}},{"id":"cache-stampede","at":"12:10"}],"eventPool":["db-failover","replication-lag"]},{"id":"hotel-booking","title":"Hotel booking","difficulty":"Medium","concepts":["search-indexing","consistency","rate-limiting"],"like":"a site like Booking.com","brief":"You run a hotel booking site. People search for rooms, open hotel pages and book. Search can be a little behind, but you must never sell the last room twice. Bots copy your prices all day, and sometimes the payment company slows down.","group":"travel","daily":true,"requires":{"payment":[["payment-processor"]]},"palette":null,"events":[{"id":"bot-scraping","at":"10:00"},{"id":"slow-dependency","at":"15:30"},{"id":"replication-lag","at":"20:15","params":{"lagSec":45}}],"eventPool":["viral-spike","cache-stampede","zone-outage"]},{"id":"airline-booking","title":"Airline booking","difficulty":"Hard","concepts":["consistency","circuit-breakers","security-compliance"],"like":"an airline's booking and seat system","brief":"You run the booking system for an airline. Fare searches are heavy, seats are limited, and every flight needs the right number of tickets, no more and no fewer. You depend on a partner system for fares from other airlines, and it has strict limits. A fare sale and a slow partner will both happen on the same day.","group":"travel","daily":true,"requires":{"payment":[["payment-processor"]]},"palette":null,"events":[{"id":"slow-dependency","at":"09:30"},{"id":"flash-crowd","at":"12:00","params":{"factor":6,"rampSec":120,"holdSec":1200,"decaySec":1800,"types":["search","read","login"]}},{"id":"third-party-rate-limit","at":"19:00"}],"eventPool":["dependency-outage","zone-outage","compliance-audit","replication-lag"]},{"id":"concert-ticket-drop","title":"Concert ticket drop","difficulty":"Hard","concepts":["queues-backpressure","rate-limiting","idempotency-retries"],"like":"a ticket sale like Ticketmaster","brief":"Tickets for a huge tour go on sale at 10:00 sharp. Hundreds of thousands of fans arrive in the same minute, bots arrive earlier, and everyone refreshes when the page stalls. There are 40,000 seats. Sell each one once, charge each buyer once, and keep the site standing.","group":"travel","daily":true,"requires":{"payment":[["payment-processor"]]},"palette":null,"events":[{"id":"bot-scraping","at":"09:50"},{"id":"flash-crowd","at":"10:00","params":{"factor":25,"rampSec":30,"holdSec":900,"decaySec":1800}},{"id":"retry-storm","at":"10:07"}],"eventPool":["slow-dependency","dependency-outage","duplicate-delivery"]},{"id":"ev-charging-network","title":"EV charging network","difficulty":"Easy","concepts":["idempotency-retries","queues-backpressure"],"like":"a public charger network","brief":"Thousands of chargers report their status every few seconds and send a record when a session ends. Drivers open an app to find a free charger. A session record must never be lost or counted twice, because it becomes a bill.","group":"iot","daily":true,"requires":{},"palette":null,"events":[{"id":"dependency-outage","at":"17:30"},{"id":"thundering-herd","at":"08:00","params":{"factor":4,"types":["telemetry"]}}],"eventPool":["retry-storm","duplicate-delivery"]},{"id":"smart-home-hub","title":"Smart home","difficulty":"Medium","concepts":["realtime-connections","multi-region","queues-backpressure"],"like":"lights and locks controlled from a phone","brief":"One million homes have a hub that keeps a live connection to your cloud. The hub reports its devices every few minutes, and the phone app sends commands like \"lock the door\". Commands must feel instant and must not be repeated. Once in a while a power cut ends and every hub reconnects together.","group":"iot","daily":true,"requires":{},"palette":null,"events":[{"id":"thundering-herd","at":"08:00","params":{"factor":8,"types":["telemetry","message","read"]}},{"id":"region-outage","at":"20:00"}],"eventPool":["duplicate-delivery","network-partition","dependency-outage"]},{"id":"smart-power-grid","title":"Smart power grid","difficulty":"Medium","concepts":["storage-tiers","queues-backpressure","security-compliance"],"like":"a power company reading millions of smart meters","brief":"A power company has 5 million smart meters. Each reports usage every 15 minutes, and they all try on the quarter hour. Customers see their usage, billing needs exact totals for seven years, and a regulator checks that records are kept safely. Control commands to home batteries and thermostats must never run twice.","group":"iot","daily":true,"requires":{},"palette":null,"events":[{"id":"network-partition","at":"09:20"},{"id":"thundering-herd","at":"16:45","params":{"factor":3,"types":["telemetry","event"]}},{"id":"compliance-audit","at":"21:00"}],"eventPool":["disk-full","duplicate-delivery","data-corruption"]},{"id":"industrial-sensors","title":"Factory sensors","difficulty":"Medium","concepts":["storage-tiers","observability"],"like":"sensors on a factory floor","brief":"A factory has thousands of machines, each sending readings many times a second. Engineers open charts of the last hour and last month, and an alarm must fire within seconds when a reading goes out of range. The data grows without end.","group":"iot","daily":true,"requires":{},"palette":null,"events":[{"id":"thundering-herd","at":"08:00","params":{"factor":3,"types":["telemetry"]}},{"id":"network-partition","at":"15:00"}],"eventPool":["disk-full","cardinality-explosion"]},{"id":"firmware-rollout","title":"Firmware rollout","difficulty":"Hard","concepts":["cdn","rate-limiting"],"like":"an over-the-air update for a million smart devices","brief":"A new firmware goes out to a million smart devices at 03:00 when they are idle. Every device asks \"is there an update?\", then downloads the same file in pieces, then reports back. If they all wake together, they can flatten your service, and a device that fails to update must not be left broken.","group":"iot","daily":false,"requires":{},"palette":null,"events":[{"id":"thundering-herd","at":"03:05","params":{"factor":3}},{"id":"network-partition","at":"14:00"}],"eventPool":["cache-stampede","zone-outage"]},{"id":"semantic-search","title":"Semantic search","difficulty":"Easy","concepts":["ai-serving","search-indexing"],"like":"\"search by meaning\" on a help center","brief":"A help center lets people type a question in plain words and finds the right articles even when the words differ. Each question is turned into a list of numbers (a vector) and matched to similar articles. Many people ask the same few questions.","group":"ai","daily":true,"requires":{},"palette":null,"events":[{"id":"viral-spike","at":"10:00","params":{"factor":4}},{"id":"bad-deploy","at":"14:00"}],"eventPool":["gpu-shortage","cache-stampede"]},{"id":"recommendations","title":"Recommendations","difficulty":"Medium","concepts":["caching","ai-serving"],"like":"a \"you may also like\" row","brief":"Every page shows a row of things the visitor might like. A ranking model builds each row, but the model is slow and costly. The row must appear instantly, even when the model is having a bad day.","group":"ai","daily":true,"requires":{},"palette":null,"events":[{"id":"slow-dependency","at":"12:00","params":{"target":"el:recommender"}},{"id":"cache-stampede","at":"20:00"},{"id":"replication-lag","at":"20:30"}],"eventPool":["viral-spike","bad-deploy"]},{"id":"ad-serving","title":"Ad serving","difficulty":"Hard","concepts":["caching","cost","ai-serving"],"like":"ad auctions on a news site","brief":"Each page view asks you to pick the best ad in about a quarter of a second: check who is allowed to see what, score the candidates with a model, run a quick auction and answer. Every impression and click must be counted exactly because money depends on it.","group":"ai","daily":true,"requires":{},"palette":null,"events":[{"id":"viral-spike","at":"11:00","params":{"factor":4}},{"id":"slow-dependency","at":"14:00","params":{"target":"el:model-server"}},{"id":"cost-blowout","at":"16:00"},{"id":"compliance-audit","at":"20:00","params":{"checks":["consent","encryption"]}}],"eventPool":["cache-stampede","bad-deploy"]},{"id":"ai-chatbot-serving","title":"AI chatbot serving","difficulty":"Hard","concepts":["ai-serving","cost","rate-limiting"],"like":"a chat assistant like ChatGPT","brief":"Millions of people ask your assistant questions. Each reply takes a GPU several seconds, and GPUs are expensive and sometimes simply unavailable. Keep replies flowing, protect your wallet from abuse, and have a plan for when the GPUs run out.","group":"ai","daily":true,"requires":{},"palette":null,"events":[{"id":"gpu-shortage","at":"10:00","params":{"factor":1.8}},{"id":"third-party-rate-limit","at":"14:00"},{"id":"cost-blowout","at":"17:00"}],"eventPool":["viral-spike","bad-deploy"]},{"id":"leaderboards","title":"Leaderboards","difficulty":"Easy","concepts":["hot-keys","caching"],"like":"a global high-score table","brief":"Players send their score at the end of every match and open the top-100 table all day. Everyone looks at the same top 100, which makes that one answer extremely popular. A tournament final will make it much worse.","group":"games","daily":true,"requires":{},"palette":null,"events":[{"id":"hot-key","at":"20:00","params":{"share":0.5}},{"id":"viral-spike","at":"20:30","params":{"factor":5}}],"eventPool":["cache-stampede","replication-lag"]},{"id":"matchmaking","title":"Multiplayer matchmaking","difficulty":"Medium","concepts":["queues-backpressure","multi-region"],"like":"finding a match in an online game","brief":"Players press Play and wait. You must put them in a match with others nearby, with similar skill, in a few seconds. Players in a far-away region will have a bad time, so you must keep each region close.","group":"games","daily":true,"requires":{},"palette":null,"events":[{"id":"overseas-launch","at":"17:00","params":{"region":"apac","share":0.4}},{"id":"flash-crowd","at":"20:00","params":{"factor":3,"holdSec":1800}},{"id":"zone-outage","at":"21:30"}],"eventPool":["region-outage","thundering-herd"]},{"id":"game-patch-day","title":"Patch day downloads","difficulty":"Medium","concepts":["cdn","cost"],"like":"a big game update","brief":"A popular game ships a 5 GB update at 10 am. Millions of players press Update within the same hour. The files are identical for everyone, so the work is moving huge amounts of data without a huge bill, while players still log in and play.","group":"games","daily":true,"requires":{},"palette":null,"events":[{"id":"flash-crowd","at":"10:00","params":{"factor":3}},{"id":"cost-blowout","at":"12:00"}],"eventPool":["cache-stampede","zone-outage"]},{"id":"battle-royale","title":"Battle royale launch","difficulty":"Hard","concepts":["realtime-connections","hot-keys"],"like":"a 100-player last-one-standing game on launch night","brief":"On launch night millions of players queue for matches of 100. Each match streams positions many times a second, the lobby shows the live player count, and one patch notes page is opened by everyone. Connections must stay open, a match server dying must not end a match, and a few hot items, like the daily reward, are hit by everybody.","group":"games","daily":false,"requires":{},"palette":null,"events":[{"id":"viral-spike","at":"20:00","params":{"factor":2}},{"id":"hot-key","at":"20:30","params":{"share":0.3}}],"eventPool":["zone-outage","cache-stampede"]},{"id":"calendar-invites","title":"Calendar and invites","difficulty":"Easy","concepts":["queues-backpressure","idempotency-retries"],"like":"a shared calendar","brief":"People view their week, create meetings and invite others. Each invite sends an email or phone alert. Monday at 9 am everyone opens the calendar at once. A repeated send must never produce two invites, and a slow email partner must not slow the calendar.","group":"productivity","daily":true,"requires":{},"palette":null,"events":[{"id":"third-party-rate-limit","at":"09:00","params":{"rpsCapFactor":0.4,"durationSec":1200}},{"id":"duplicate-delivery","at":"13:00","params":{"share":0.05}}],"eventPool":["cache-stampede","slow-dependency"]},{"id":"file-sync","title":"File sync","difficulty":"Medium","concepts":["storage-tiers","consistency"],"like":"a file sync app like Dropbox","brief":"A folder on your laptop mirrors to the cloud and to your other devices. Small changes must arrive quickly, big files must upload reliably, and a shared folder with thousands of people must not melt the metadata service.","group":"productivity","daily":true,"requires":{},"palette":null,"events":[{"id":"thundering-herd","at":"09:00","params":{"factor":6,"types":["sync","read"]}},{"id":"hot-key","at":"13:00","params":{"share":0.3}},{"id":"disk-full","at":"15:00"}],"eventPool":["network-partition","replication-lag"]},{"id":"collaborative-docs","title":"Collaborative documents","difficulty":"Hard","concepts":["realtime-connections","consistency"],"like":"shared docs like Google Docs","brief":"Many people edit the same document at the same moment and everyone must see a consistent result in a fraction of a second. Edits must never be lost, even if a server crashes in the middle of typing or two people type in the same place.","group":"productivity","daily":true,"requires":{},"palette":null,"events":[{"id":"thundering-herd","at":"09:00","params":{"factor":5,"types":["read","sync","message"]}},{"id":"zone-outage","at":"11:00"},{"id":"network-partition","at":"14:30"}],"eventPool":["region-outage","schema-migration-lock"]},{"id":"email-service","title":"Email service","difficulty":"Hard","concepts":["search-indexing","storage-tiers","queues-backpressure"],"like":"an email service like Gmail","brief":"Mail arrives from the whole internet, including a lot of spam, and must land in the right inbox. People open, search, send and attach. Never lose a message, and keep search fast even over years of mail.","group":"productivity","daily":true,"requires":{},"palette":null,"events":[{"id":"bot-scraping","at":"10:00"},{"id":"disk-full","at":"14:00"},{"id":"replication-lag","at":"18:00"}],"eventPool":["viral-spike","thundering-herd"]},{"id":"shopping-cart","title":"Shopping cart","difficulty":"Easy","concepts":["caching","consistency"],"like":"the cart on an online store like Amazon","brief":"Shoppers browse products, drop things in a cart and come back later from another phone. Browsing is most of the traffic and can be a little old. The cart itself must always show what the shopper just added.","group":"commerce","daily":true,"requires":{},"palette":null,"events":[{"id":"flash-crowd","at":"12:00","params":{"factor":2.5}},{"id":"cache-stampede","at":"20:00"}],"eventPool":["slow-dependency"]},{"id":"flash-sale-inventory","title":"Flash sale inventory","difficulty":"Medium","concepts":["hot-keys","queues-backpressure"],"like":"a limited-stock sale like a sneaker drop","brief":"At noon a hot item goes on sale with only a few thousand units. A huge crowd arrives in the same minute and everyone taps Buy. Stock must never go below zero, nobody should be charged for an item that is gone, and the site must not fall over.","group":"commerce","daily":true,"requires":{},"palette":null,"events":[{"id":"thundering-herd","at":"12:00","params":{"factor":3,"types":["read"]}},{"id":"bot-scraping","at":"12:05"}],"eventPool":["hot-key","db-failover"]},{"id":"otp-sms-delivery","title":"One-time code by text","difficulty":"Easy","concepts":["idempotency-retries","rate-limiting"],"like":"the login code text that arrives in seconds","brief":"Apps ask you to send a six-digit code to a phone. The code must arrive in seconds, a person must never get it twice, and nobody may use your system to spam a number. Texts go through a carrier partner with a hard speed limit and a real price per message.","group":"telecom","daily":true,"requires":{},"palette":null,"events":[{"id":"third-party-rate-limit","at":"09:00","params":{"rpsCapFactor":0.8}},{"id":"duplicate-delivery","at":"13:00","params":{"share":0.05}}],"eventPool":["retry-storm","dependency-outage"]},{"id":"call-routing","title":"Voice and video call setup","difficulty":"Medium","concepts":["realtime-connections","multi-region"],"like":"the ringing part of a calling app or phone carrier","brief":"People tap a name and expect the other phone to ring in a second or two. Your job is the setup: find where the person is, ring their devices and connect the two sides. Everyone calls at the same moments, like evenings and holidays.","group":"telecom","daily":true,"requires":{},"palette":null,"events":[{"id":"flash-crowd","at":"19:00","params":{"factor":2.5}},{"id":"zone-outage","at":"21:00"}],"eventPool":["network-partition","slow-dependency"]},{"id":"status-page","title":"Status page","difficulty":"Easy","concepts":["multi-region","cdn"],"like":"a public status page","brief":"When a big service has a problem, millions of people open its status page to see what is happening. The page must stay up exactly when everything else is down, and it must not share a home with the thing it reports on. A few staff update it by hand.","group":"devtools","daily":true,"requires":{},"palette":null,"events":[{"id":"flash-crowd","at":"14:00","params":{"factor":5}},{"id":"zone-outage","at":"14:20"}],"eventPool":["region-outage","cache-stampede"]},{"id":"ci-cd-pipeline","title":"CI/CD build system","difficulty":"Medium","concepts":["queues-backpressure","cost","autoscaling"],"like":"build runners like GitHub Actions","brief":"Every time a developer saves code to the shared repository, you build and test it. Thousands of developers do this all day, and release day brings a rush. Builds must start quickly, but idle build machines cost real money.","group":"devtools","daily":true,"requires":{},"palette":null,"events":[{"id":"flash-crowd","at":"10:00","params":{"factor":6,"rampSec":60,"holdSec":1800}},{"id":"poison-message","at":"14:00"},{"id":"cost-blowout","at":"17:00"}],"eventPool":["bad-deploy","disk-full"]},{"id":"feature-flag-service","title":"Feature flag service","difficulty":"Medium","concepts":["realtime-connections","cdn"],"like":"a flag service like LaunchDarkly","brief":"Companies turn features on and off for groups of users without redeploying. Millions of apps ask \"is this on for me?\", and the answer must be instant. A wrong flag value can take down an entire customer, so changes must roll out safely and be reversible in seconds.","group":"devtools","daily":true,"requires":{},"palette":null,"events":[{"id":"thundering-herd","at":"09:00","params":{"factor":6,"types":["read"]}},{"id":"bad-config-push","at":"11:00"}],"eventPool":["cache-stampede","zone-outage"]},{"id":"clickstream-analytics","title":"Clickstream analytics","difficulty":"Medium","concepts":["queues-backpressure","storage-tiers"],"like":"the page-view tracking behind a product analytics tool","brief":"Every tap and page view on a busy site sends a tiny event to you. Hundreds of millions pile up each day. Analysts open dashboards to ask what happened this hour. Never drop an event, and keep the storage bill under control as the pile grows.","group":"devtools","daily":true,"requires":{},"palette":null,"events":[{"id":"viral-spike","at":"20:00","params":{"factor":2}},{"id":"cardinality-explosion","at":"14:00"},{"id":"cost-blowout","at":"16:00"}],"eventPool":["poison-message","disk-full"]},{"id":"observability-platform","title":"Metrics platform","difficulty":"Hard","concepts":["sharding","observability","multi-region"],"like":"a metrics store like Mimir","brief":"Thousands of services send a stream of measurements: CPU, errors, latency. You store them, draw the dashboards and fire the alerts. The platform watching everyone else must never go down first, and one customer creating millions of new series can ruin the bill.","group":"devtools","daily":true,"requires":{},"palette":null,"events":[{"id":"cardinality-explosion","at":"08:00"},{"id":"thundering-herd","at":"09:00","params":{"factor":8,"types":["report","read"]}},{"id":"zone-outage","at":"12:00"}],"eventPool":["noisy-neighbor","disk-full"]}],"lessons":[{"id":"intro","title":"Your first system","track":"Start","minutes":3,"summary":"Build a tiny app in three minutes and keep it standing through a rush of visitors.","takeaway":"A system is a few boxes in a line, and design is deciding which boxes and how many. Count first: the first practice problem, a URL shortener at about 5,000 requests a second, needs about 6 servers, not 3 (approx.).","icon":null},{"id":"requests-and-latency","title":"What a request is, and p50 versus p99","track":"Start","minutes":3,"summary":"A request is one question and one answer. Averages hide the slow ones that people remember.","takeaway":"Judge a system by its slow requests, and remember distance is part of latency.","icon":"ui.clock"},{"id":"back-of-envelope","title":"Numbers you can do in your head","track":"Start","minutes":3,"summary":"Turn monthly users into requests per second and servers needed, with a pencil.","takeaway":"Divide by 100,000 for a day, multiply by 3 for the peak, and leave headroom.","icon":"ui.budget"},{"id":"load-balancing","title":"Spreading load","track":"Scale the front","minutes":3,"summary":"A load balancer hands each request to a free server so none is overwhelmed.","takeaway":"A load balancer lets many servers act as one, and hides the ones that fail.","icon":"el.regional-lb"},{"id":"caching","title":"Remembering answers","track":"Scale the front","minutes":3,"summary":"A cache keeps recent answers in memory so the database is asked less.","takeaway":"A cache trades a little freshness for a lot of speed and a calmer database.","icon":"el.cache"},{"id":"cdn","title":"Copies near users","track":"Scale the front","minutes":3,"summary":"A CDN keeps copies of files and pages close to people, so your servers see far fewer requests.","takeaway":"Put copies of shared content near people; keep your servers for what only they can do.","icon":"el.cdn"},{"id":"autoscaling","title":"Adding servers, and why it is late","track":"Scale the front","minutes":3,"summary":"Autoscaling adds servers when busy, but new servers take minutes to start.","takeaway":"Autoscaling follows the crowd; for known surges, scale ahead of it.","icon":"el.autoscaler"},{"id":"replication-lag","title":"Copies are a little behind","track":"Scale the data","minutes":3,"summary":"Read copies share the load, but they trail the original by a moment.","takeaway":"Replicas scale reads; remember they are always slightly behind.","icon":"el.sql-replica"},{"id":"sharding","title":"Splitting data","track":"Scale the data","minutes":3,"summary":"When one database cannot take the writes, split the data across several.","takeaway":"Shard when writes or size outgrow one machine, and pick a key that spreads evenly.","icon":"el.sharded-sql"},{"id":"hot-keys","title":"When one item is popular","track":"Scale the data","minutes":3,"summary":"Sharding spreads keys, not popularity. One star item can still overload one shard.","takeaway":"Spread keys with shards and spread attention with caches and copies.","icon":"ev.hot-key"},{"id":"search-indexing","title":"Search is a copy","track":"Scale the data","minutes":3,"summary":"A search index is a second copy of your data, shaped for finding words fast.","takeaway":"Keep one source of truth and feed purpose-built copies for search.","icon":"el.search-index"},{"id":"storage-tiers","title":"Hot, warm, cold and archive","track":"Scale the data","minutes":3,"summary":"Old data is rarely read, so keep it somewhere cheaper and slower.","takeaway":"Match storage to how often data is read, and move it down as it ages.","icon":"el.cold-archive"},{"id":"queues-backpressure","title":"Waiting instead of breaking","track":"Work later","minutes":3,"summary":"A queue lets a burst wait in line instead of crashing the thing behind it.","takeaway":"Queues turn spikes into waiting; backpressure stops the waiting from growing forever.","icon":"el.queue"},{"id":"fanout","title":"One post, millions of feeds","track":"Work later","minutes":3,"summary":"A post by a celebrity must reach millions of feeds; do it in the background.","takeaway":"Do heavy fan-out in the background, and treat huge accounts differently.","icon":"ev.celebrity-fanout"},{"id":"realtime-connections","title":"Live sockets and presence","track":"Work later","minutes":3,"summary":"Chat and live scores keep a connection open; that changes how you count capacity.","takeaway":"Count open connections, not just requests, and keep presence light.","icon":"el.websocket-gateway"},{"id":"consistency","title":"Which copy is the truth","track":"Stay correct","minutes":3,"summary":"With several copies, you choose between always-latest and always-fast.","takeaway":"Pick the strongest guarantee only where being wrong would hurt.","icon":"ev.replication-lag"},{"id":"idempotency-retries","title":"Exactly once, on purpose","track":"Stay correct","minutes":3,"summary":"Retries are normal, so every action must be safe to repeat.","takeaway":"Give every action a key, and repeating it becomes harmless.","icon":"el.idempotency-store"},{"id":"rate-limiting","title":"Saying \"slow down\" politely","track":"Stay up","minutes":3,"summary":"A limit per client stops one noisy caller or a flood from taking everyone down.","takeaway":"Limit each caller so one flood cannot become everyone's outage.","icon":"el.rate-limiter"},{"id":"circuit-breakers","title":"Timeouts and failing fast","track":"Stay up","minutes":3,"summary":"When a dependency is slow, give up early so your servers do not pile up waiting.","takeaway":"Wait briefly, stop calling what is failing, and have a fallback.","icon":"el.circuit-breaker"},{"id":"multi-region","title":"Surviving a region","track":"Stay up","minutes":4,"summary":"A whole region can go dark. Run in two, and send people to the one that is up.","takeaway":"Multiple regions buy survival at the price of cost and harder data.","icon":"ev.region-outage"},{"id":"observability","title":"Seeing problems early","track":"Stay up","minutes":3,"summary":"You cannot fix what you cannot see: metrics, alerts and a way back.","takeaway":"Watch what users feel, tell a human fast, and keep a way back.","icon":"el.metrics"},{"id":"cost","title":"Where the money goes","track":"Business","minutes":3,"summary":"Every box has a monthly price. Good design meets the goal without waste.","takeaway":"Meet the goal with the least you can; money is a requirement too.","icon":"ui.cost"},{"id":"security-compliance","title":"Proof built in","track":"Business","minutes":3,"summary":"Auditors want proof: who did what, who can see what, and that data is protected.","takeaway":"Build the proof in from the start; adding it after the audit is far harder.","icon":"el.audit-log"},{"id":"ai-serving","title":"GPUs, tokens and fallbacks","track":"Business","minutes":3,"summary":"Serving AI answers is slow and expensive, so cache, limit, and have a fallback.","takeaway":"Cache, cap and fall back; GPUs are the scarcest part of the system.","icon":"el.model-server"}]}