Changing a running system

The new column is ready, but old apps are not

You want to replace a free-text address with structured fields. Web code can update today, but mobile clients will keep sending the old shape for months. A migration is a period when multiple schemas and code versions coexist. A deployment that works only after every client updates is not a rollout plan.

Expand, move, contract

Advance a live table through compatible schema expansion, dual writes, backfill, comparison, cutover and contraction. Inject a stage failure, keep writing, then inspect the rollback boundary.

Worked example: how long it takes

Retries, live changes, reconciliation, indexes and replicas add work beyond these lower bounds. A canary might admit 1%, 5%, 25% and then all eligible traffic. Percentages do not replace minimum observation time or coverage of large tenants. The traffic engine does not execute schema changes row by row.

Copy time and catch-up time differ

• Backfill: 27.8 hours: 500 million rows / 5,000 rows/s. About 250 GB at 500 bytes/row.
• Catch up: 2.8 hours: 30 million changes / (5,000 − 2,000 live changes/s).
• Verify before routing: Include minimum observation time and large-tenant coverage in the canary.

Rollback is a data decision

Define which reader versions understand new writes and what happens to writes accepted during a canary. A code rollback is unsafe when old readers cannot understand the new representation. Preserve compatible writes or test a reverse transform. Agree ownership with application and data teams, on-call, security, product and support. Calendar time can be weeks even when copying bytes takes a day.

Rollback follows the data

• Preserve old readers: Keep compatible writes during backfill and the canary.
• Watch stop thresholds: Mismatch rate, lag, p99 and lock time can pause the rollout.
• Contract only after proof: Dropping old data ends a code-only rollback. Reserve a stable rollback period first.

Key numbers

Backfill floor

≈ 27.8 hours. 500 million ÷ 5,000/s

Logical copied bytes

≈ 250 GB. 500 million × 500 B, before index and replication

Change catch-up

≈ 2.8 hours. 30 million ÷ (5,000 - 2,000)/s

Check your reasoning

Think through these questions before opening the lesson. In the game you can test the idea and compare your reasoning.

• A backfill reads version 5, then a live write creates version 6. What prevents the delayed copy from regressing the target?
• Why not write to both databases in the request handler? Follow-up: the first write succeeds and the second times out.
• A canary writes a new enum that old code cannot parse. What must be addressed before reverting the binary?
• The backfill takes one day. Why did you ask for three weeks? Follow-up: who can stop the rollout?
• A 1% canary passes for thirty seconds on a low-volume service. What evidence is still missing?