Tracing platform

The brief

Store distributed traces for 500 teams and make an exemplar link open the useful trace during an incident. Decide what to sample before storage, handle late spans, and keep query traffic from overwhelming ingestion. Keeping all errors is a policy goal with a failure boundary, not a magic property of sampling.

Think of a trace store like Tempo.

Users, scale and requirements

Who uses it

500 teams and their services (approx.)

Audience in plain numbers

10,000 users or devices

Normal peak

≈ 100 requests per second

Request mix

Batch task: 0.1%; Read: 99.9%

Speed for 99 of 100 requests

Read: ≈ 1,000 ms

Successful requests

99.9%

Data safety

No acknowledged write may be lost.

Monthly budget

≈ $2,400

Data at rest

≈ 6.048 TB

Where users are

North America: 100%

Home region

North America

Waiting deadlines

Batch task: 3,600 seconds; Read: 900 seconds

Minimum retained history

7 days on authoritative stores

≈ marks an approximation, just as in the game. Waiting deadlines measure the age of queued work and are separate from interactive response time.

• Collectors see 10,000 traces/s with 20 spans/trace and 500 B/span. Retain 10% for seven days; a raw tail buffer holds spans for an assumed average of 30 seconds before selection (approx.).
• One catalog batch is a 100 MB retained block. Raw collection and the selected-block workload have separate capacity boundaries.
• The reference starts with already selected and formed blocks. The API acknowledges only after object storage accepts them durably. Raw-span collection, sampling, durable collector spooling and index publication need separate capacity and failure tests.
• Sources emit only their assigned request types. The source rates below reproduce the aggregate workload; protocol guarantees still require the separate design checkpoint.
• Scored data retention: at least 7 days on active authoritative stores. Copies used only for search or reports can be rebuilt. Local device history and rollup policies still require the separate protocol check.

Questions that shape the design

What reaches the sampler?

About 10,000 traces/s, 20 spans/trace and 500 bytes/span before compression. If average span residence is 30 seconds, budget about 3 GB of raw payload before overhead and redundancy. A 30-second decision timer alone does not establish residence time; span arrival patterns and decision caches also matter (approx.).

Must we keep every trace?

No. Plan a representative 10% baseline and prioritize error and slow traces subject to a hard byte budget. The modeled storage load is the 10% baseline. An error-heavy incident can exceed it, so reserve capacity or explicitly shed lower-priority samples.

How are late spans handled?

Route a trace id to one decision group, persist decisions for a bounded late-span window and append late spans consistently where possible. Mark incomplete traces. Collector loss or upstream sampling can still remove evidence.

What does the score measure?

Retained 100 MB blocks and small trace-id reads after sampling. Tail assembly, outcome selection and full attribute searches are worked obligations; placing the Traces observer does not implement a tracing platform.

The traffic day

Fairly steady traffic throughout the day. The clock covers 00:00 to 24:00 in the home region. Events add pressure on top of that normal traffic.

1. 11:00 · Zone outage

   An availability zone loses power or network access.

   Duration: ≈ 3,600 seconds.

2. 14:00 · Synchronized rush

   Many callers send requests together instead of spreading them over time.

   Traffic for Read reaches 4 times the scheduled load; rises over 10 seconds, holds for 60 seconds and eases over 300 seconds. Other request types keep their scheduled load.

3. 18:30 · Retry storm

   A brief failure causes clients to retry, adding load while the service recovers.

   Failed calls: 90%. Duration: ≈ 30 seconds. Duration: ≈ 1,200 seconds.

What a strong design has to handle

• Does the sampling decision need to know whether the request eventually failed?
• Make tail sampling decisions within a bounded buffer and explain late-span loss.
• Head sampling can discard the trace before an error occurs. Select consistently by trace id so services do not independently retain fragments.
• Tail selection receives the raw stream before choosing. At an assumed 30-second average residence, this example buffers about 3 GB raw payload; arrival timing, process overhead, replicas, decision caches and long traces change that estimate.

Make a prediction, run a full traffic day, then compare the result. The in-game editorial opens after a finished day and includes measured reference designs.

Concepts to practise

• Logs and traces
• Ordering and time
• Consistent hashing and moving data
• Inside a storage engine
• SLOs and error budgets