AI chatbot serving

The brief

Millions of people ask your assistant questions. Each reply takes a GPU several seconds, and GPUs are expensive and sometimes simply unavailable. Keep replies flowing, protect your wallet from abuse, and have a plan for when the GPUs run out.

Think of a chat assistant like ChatGPT.

Users, scale and requirements

Who uses it

8 million people a month

Audience in plain numbers

8,000,000 people per month

Normal peak

≈ 800 requests per second

Request mix

AI request: 15%; Read: 70%; Write: 15%

Speed for 99 of 100 requests

AI request: ≈ 14,000 ms; Read: ≈ 300 ms; Write: ≈ 300 ms

Successful requests

99.5%

Data safety

At most 1 minute of writes may be lost.

Monthly budget

≈ $130,000

Data at rest

≈ 20 TB

Where users are

North America: 60%; Europe: 40%

Home region

North America

Waiting deadlines

AI request: 900 seconds; Read: 900 seconds; Write: 3,600 seconds

Fresh reads

People must see their own changes. At most 0.1% of reads may be stale in the model.

Personal reads

90% of reads are private; a shared public copy cannot answer them.

Required proofs

Managed encryption keys; Retention rules

≈ marks an approximation, just as in the game. Waiting deadlines measure the age of queued work and are separate from interactive response time.

• A GPU makes about 6 replies per second with batching (approx.).
• A hosted model costs about $0.003 per reply (approx.).

Questions that shape the design

How long may a reply take?

The first words should appear in about a second. A complete reply may take up to about 14 seconds; during a GPU shortage a few take longer.

What if we run out of GPUs?

Use a hosted model from an outside provider as overflow, at a higher cost per reply. Better a pricier answer than an error.

Must conversations be stored?

Yes, so people can come back to them. They are private and encrypted.

Should every question be answered fresh?

No. Very common questions can reuse a saved answer.

The traffic day

Consumer traffic, busiest in the evening. The clock covers 00:00 to 24:00 in the home region. Events add pressure on top of that normal traffic.

1. 10:00 · GPU shortage

   Demand rises while the cloud cannot supply more GPUs.

   Traffic for AI request reaches 1.8 times the scheduled load; rises over 900 seconds, holds for 7,200 seconds and eases over 1,800 seconds. Other request types keep their scheduled load. Duration: ≈ 7,200 seconds.

2. 14:00 · Provider rate limit

   An outside service lowers its configured request quota and answers 'too many requests' above the reduced limit.

   Provider capacity falls to 50% of its configured quota. Duration: ≈ 3,600 seconds.

3. 17:00 · Cost blowout

   A crawler hits uncached pages all day, and every pay-per-use part of the bill multiplies.

   Traffic for Read, Search reaches 1.5 times the scheduled load; rises over 600 seconds, holds for 21,600 seconds and eases over 600 seconds. Other request types keep their scheduled load. Billed usage multiplier: 3. Duration: ≈ 21,600 seconds.

What a strong design has to handle

• GPUs are the scarce, expensive part. What can stand in front of them to absorb abuse and repeats?
• Admit generation within token and GPU budgets and recover interrupted replies explicitly.
• A semantic cache saves GPU time on repeats but can return a slightly different answer to a similar question.
• Own GPUs cost less per reply at steady load. A hosted model costs more per reply but needs no buying ahead.

Make a prediction, run a full traffic day, then compare the result. The in-game editorial opens after a finished day and includes measured reference designs.

Concepts to practise

• GPUs, tokens and fallbacks
• Where the money goes
• Saying "slow down" politely