Skip to content
System Designer

Medium · Core building blocks

Rate limiter for a public API

Developers call your public API from scripts and apps. A few misbehaving or malicious clients can crowd out everyone else. Build limits that are fair, fast and cheap, and that survive a retry storm. Practise this medium system design problem for free with a Google or GitHub account.

The brief

Developers call your public API from scripts and apps. A few misbehaving or malicious clients can crowd out everyone else. Build limits that are fair, fast and cheap, and that survive a retry storm.

Think of the limits on a public API like GitHub's.

Users, scale and requirements

Who uses it
2 million developers
Audience in plain numbers
2,000,000 users or devices
Normal peak
≈ 20,000 requests per second
Request mix
Read: 85%; Write: 15%
Speed for 99 of 100 requests
Read: ≈ 250 ms; Write: ≈ 250 ms
Successful requests
99.9%
Data safety
At most 1 minute of writes may be lost.
Monthly budget
≈ $89,000
Data at rest
≈ 1 TB
Where users are
North America: 60%; Europe: 40%
Home region
North America
Waiting deadlines
Read: 900 seconds; Write: 3,600 seconds

≈ marks an approximation, just as in the game. Waiting deadlines measure the age of queued work and are separate from interactive response time.

  • Estimate memory for 2 million counters at about 100 bytes per key, including overhead.
  • Typical limit: 5,000 requests an hour per key, bursts allowed (approx.).

Questions that shape the design

What should an over-limit client see?

A quick, polite "429 slow down" with a hint when to try again. It must be faster than a normal answer.

Is the limit per person or per key?

Per API key. One customer may have many servers sharing one key.

If the limiter itself breaks, do we let traffic through or block it?

Let it through for ordinary traffic. A broken limiter must not take the whole API down. Abuse protection is still wanted.

How exact must the counting be?

Close is fine. Being off by a few percent is acceptable.

The traffic day

Traffic concentrated in working hours. The clock covers 00:00 to 24:00 in the home region. Events add pressure on top of that normal traffic.

  1. 10:00 · Bot scraping

    Bots try stolen passwords and scrape pages, adding unwanted traffic.

    Extra attack traffic is 1 times the configured peak. Duration: ≈ 7,200 seconds.

  2. 15:30 · Retry storm

    A brief failure causes clients to retry, adding load while the service recovers.

    Failed calls: 90%. Duration: ≈ 30 seconds. Duration: ≈ 1,200 seconds.

What a strong design has to handle

  • Where is the cheapest place to say no: before your servers do any work, or after?
  • Enforce a fair per-key limit across replicas and choose an outage policy.
  • Fail-open keeps users happy when the limiter breaks, but briefly lets abusers through. Fail-closed is safer for abuse, riskier for uptime.
  • A shared counter is exact but adds a network hop. Local counters are fastest but let a key exceed the limit across many servers.

Make a prediction, run a full traffic day, then compare the result. The in-game editorial opens after a finished day and includes measured reference designs.

Concepts to practise

Related problems