Skip to content
System Designer

Medium · Developer platforms

Feature flag service

Companies turn features on and off for groups of users without redeploying. Millions of apps ask "is this on for me?", and the answer must be instant. A wrong flag value can take down an entire customer, so changes must roll out safely and be reversible in seconds. Practise this medium system design problem for free with a Google or GitHub account.

The brief

Companies turn features on and off for groups of users without redeploying. Millions of apps ask "is this on for me?", and the answer must be instant. A wrong flag value can take down an entire customer, so changes must roll out safely and be reversible in seconds.

Think of a flag service like LaunchDarkly.

Users, scale and requirements

Who uses it
2,000 customer companies, 5 million app instances
Audience in plain numbers
5,000,000 users or devices
Normal peak
≈ 25,000 requests per second
Request mix
Read: 89.5%; Analytics event: 10%; Write: 0.5%
Speed for 99 of 100 requests
Read: ≈ 350 ms; Write: ≈ 350 ms
Successful requests
99.99%
Data safety
No acknowledged write may be lost.
Monthly budget
≈ $54,000
Data at rest
≈ 0.2 TB
Where users are
North America: 45%; Europe: 35%; Asia Pacific: 20%
Home region
North America
Waiting deadlines
Read: 900 seconds; Analytics event: 3,600 seconds; Write: 3,600 seconds
Fresh reads
People must see their own changes. At most 0.1% of reads may be stale in the model.
Personal reads
5% of reads are private; a shared public copy cannot answer them.

≈ marks an approximation, just as in the game. Waiting deadlines measure the age of queued work and are separate from interactive response time.

  • A rule file is about 50 KB for a big customer (approx.).
  • Usage events are small and can be a minute late.

Questions that shape the design

Do apps ask your servers on every check?

No. Apps download the rules and decide locally. Your service sends the rules and receives usage reports.

How quick must an emergency off-switch be?

Within about 30 seconds everywhere.

What if your service is down?

The app keeps using its last known rules. You must never be the cause of a customer outage.

The traffic day

Traffic concentrated in working hours. The clock covers 00:00 to 24:00 in the home region. Events add pressure on top of that normal traffic.

  1. 09:00 · Synchronized rush

    Many callers send requests together instead of spreading them over time.

    Traffic for Read reaches 6 times the scheduled load; rises over 10 seconds, holds for 60 seconds and eases over 300 seconds. Other request types keep their scheduled load.

  2. 11:00 · Bad config push

    A bad setting reaches every server at once. Everything starts failing together.

    Lasts until detected and repaired.

What a strong design has to handle

  • Every app downloads the same file for its customer. Where can a copy of that file live?
  • Evaluate consistent versioned flags and roll back unsafe changes quickly.
  • A shorter CDN keep time makes changes appear sooner but sends more traffic to the origin.
  • Pushing changes over sockets is fast but means holding millions of open connections.

Make a prediction, run a full traffic day, then compare the result. The in-game editorial opens after a finished day and includes measured reference designs.

Concepts to practise

Related problems