Skip to content
System Designer

Hard · Developer platforms

Metrics platform

Thousands of services send a stream of measurements: CPU, errors, latency. You store them, draw the dashboards and fire the alerts. The platform watching everyone else must never go down first, and one customer creating millions of new series can ruin the bill. Practise this hard system design problem for free with a Google or GitHub account.

The brief

Thousands of services send a stream of measurements: CPU, errors, latency. You store them, draw the dashboards and fire the alerts. The platform watching everyone else must never go down first, and one customer creating millions of new series can ruin the bill.

Think of a metrics store like Mimir.

Users, scale and requirements

Who uses it
2,000 services and their on-call teams
Audience in plain numbers
8,000 users or devices
Normal peak
≈ 14,000 requests per second
Request mix
Telemetry batch: 80%; Report: 12%; Read: 8%
Speed for 99 of 100 requests
Report: ≈ 5,000 ms; Read: ≈ 500 ms
Successful requests
99.95%
Data safety
At most 1 minute of writes may be lost.
Monthly budget
≈ $77,000
Data at rest
≈ 8 TB
Where users are
North America: 50%; Europe: 30%; Asia Pacific: 20%
Home region
North America
Waiting deadlines
Telemetry batch: 3,600 seconds; Report: 3,600 seconds; Read: 900 seconds

≈ marks an approximation, just as in the game. Waiting deadlines measure the age of queued work and are separate from interactive response time.

  • One ingest request batches 100 samples. Use the peak request rate and telemetry share above to estimate the batch rate.
  • A sample takes about 1.5 bytes after compression (approx.).

Questions that shape the design

How many measurements arrive?

About 17 million different series, each sampled every 15 seconds. That is around 1.1 million samples every second.

Is it OK to lose a few seconds of measurements?

About a minute is fine. But alerts must keep working during problems.

How long do we keep data?

Fifteen days at full detail. Older data is summarised elsewhere.

What if one team creates a million new series by accident?

Limit new series per team so one mistake cannot hurt everyone.

The traffic day

Fairly steady traffic throughout the day. The clock covers 00:00 to 24:00 in the home region. Events add pressure on top of that normal traffic.

  1. 08:00 · Metrics explosion

    A metric label creates many more distinct series, increasing the bill.

    Billed usage multiplier: 20. Duration: ≈ 21,600 seconds.

  2. 09:00 · Dashboard rush

    Teams open dashboards at nine. Read and report traffic rises to eight times normal while agents keep sending measurements.

    Traffic for Report, Read reaches 8 times the scheduled load; rises over 10 seconds, holds for 60 seconds and eases over 300 seconds. Other request types keep their scheduled load.

  3. 12:00 · Zone outage

    An availability zone loses power or network access.

    Duration: ≈ 3,600 seconds.

What a strong design has to handle

  • Writes are constant and huge, reads come in bursts when people open dashboards. Do they need the same machines?
  • Isolate tenants and bound label cardinality while preserving useful alerts.
  • The stream adds a few seconds before data is queryable, but protects against drops when the database stalls.
  • More shards means more write capacity and more copies of the cluster to run. Choosing the shard key matters.

Make a prediction, run a full traffic day, then compare the result. The in-game editorial opens after a finished day and includes measured reference designs.

Concepts to practise

Related problems