Read the canary

Play this lesson

Three readings

A new race server went out to 5% of Shard Garden's players. Priya has three readings of crashes per 1,000 sessions. Only one of them says "roll back now".

Try it

Find the real signal

Each reading compares the canary with something. Tap the one that means the release should be rolled back.

Try it in the lesson

Quick check

Why was Priya right not to act on 1 crash in 40 sessions?

  1. Too few sessions: one crash more or less changes everything
  2. One crash is never a problem
  3. 40 sessions is too many
Show the answer

Too few sessions: one crash more or less changes everything

With 40 sessions, a single crash swings the rate from 0 to 25 per 1,000. The canary needs to run longer.

A fair comparison

The third reading looked bad against yesterday: 4 crashes per 1,000 against 2. But the control, at the same hour, was at 4 too. Something made today worse for everyone, and the release was not it.

Spot the trap

Canary: 6 per 1,000. Last week: 3 per 1,000. Control right now: 6 per 1,000. What should the team do?

  1. Carry on: the canary matches the control, so the release is not the cause
  2. Roll back: crashes doubled
  3. Delete last week's numbers
Show the answer

Carry on: the canary matches the control, so the release is not the cause

The control is the same players' world at the same moment. A canary that matches it is no worse than the current version.

Roll back first

When a canary is clearly worse, the release is rolled back before anyone starts debugging. Google's SRE book puts it in order: "Stop the bleeding, restore service, and preserve the evidence for root-causing."

Quick check

When a canary is clearly worse, the team should find the bug before rolling back.

Show the answer

False

Every minute of debugging is a minute of crashes for real players. Roll back, then debug calmly with the evidence saved.

On the studio board

The studio adds a decision to the board:

Every release starts as a canary for 5% of players for an hour, compared with the other 95% at the same time. A canary clearly worse than the control is rolled back first and debugged after.

Bolt says: Next: blue-green deploys.

Fact-checked 12 October 2026