Canary releases

Play this lesson

Every player at once

Priya is about to ship a new race server. A release that goes to every player at once puts every player at risk. If the new race server has a bug that only shows up with real traffic, all of Shard Garden finds it together.

A canary

Google's SRE workbook defines canarying as "a partial and time-limited deployment of a change in a service and its evaluation."

"The part of the service that receives the change is 'the canary,' and the remainder of the service is 'the control.'" A problem hits a few players, not all of them.

Quick check

In a 5% canary, what is the control?

  1. The 95% of players still on the current version
  2. The 5% on the new version
  3. Yesterday's numbers
Show the answer

The 95% of players still on the current version

The control is the rest of the service, running the version that was already there.

Compare at the same moment

The workbook warns that "time is one of the biggest sources of change in observed metrics." Friday evening is busier than Tuesday morning. A new event changes how people play.

So the canary is compared with the control right now, not with how things looked before the release.

What to watch

For a web service, the workbook says the best signals are "likely the HTTP return codes and latency of response", and suggests starting from the service's SLIs, the numbers it promises.

For Shard Garden: crashes per 1,000 sessions, failed saves, and the p95 time to start a race.

Pick the comparison

Friday, 8 pm: the canary's failed saves are 0.4%. What should Priya compare that with?

  1. The control's failed saves, Friday at 8 pm
  2. Tuesday morning's failed saves, before the release
  3. Zero
Show the answer

The control's failed saves, Friday at 8 pm

The control at the same moment has the same traffic and the same events. Only the release differs.

Stages that grow

The workbook suggests "a canary containing multiple stages": "Once this stage passes successfully, the next stage will have a larger canary population." Shard Garden goes 5%, then 25%, then everyone.

It also says: "We strongly advise running only one canary deployment at a time."

Quick check

Why run only one canary at a time?

  1. With two out at once, a worse number could come from either change
  2. Two canaries cost twice as much
  3. Players notice two canaries
Show the answer

With two out at once, a worse number could come from either change

One canary means one difference between the canary and the control, so any change in the numbers points at it.

Not after a handful

The workbook also warns that stopping a canary "after receiving just a handful of queries doesn't provide a useful signal". A canary needs enough players to mean something.

Bolt says: Next: read a canary yourself.

Fact-checked 12 October 2026