Rollbacks

Play this lesson

Undo, fast

A rollback puts the last good version back. Google's SRE book lists the order of work in an incident: "Stop the bleeding, restore service, and preserve the evidence for root-causing." A rollback stops the bleeding.

Back or forward?

The other choice is to roll forward: write a fix and ship it.

Rolling back is usually faster and safer. The old version already ran for days with real players. A fix is brand new code, written in a hurry, while players wait.

Quick check

Failed saves jumped right after a release. A fix would take an hour to write and test. What does Priya do first?

  1. Roll back, then write the fix calmly
  2. Write the fix straight into production
  3. Wait an hour and see
Show the answer

Roll back, then write the fix calmly

Rolling back ends the problem for players in minutes. The fix can then be written and tested properly.

What a rollback can't undo

Code rolls back. Data does not. If version 2 writes saves in a new format, version 1 may not be able to read them.

Architecture's expand and contract fixes this: first add the new form beside the old one, so both versions work; remove the old form only once nobody needs it.

Phones can't be rolled back

On the web, the next page load gets the old version. A phone app is different. Google Play can halt a staged rollout so "no additional users will receive" it, but "users who already received" it "will remain on that version."

Google's advice for a broken version is to "create and roll out a new release with a fixed app bundle."

Pick the safe plan

Version 2 changes the save format. Which plan keeps a rollback possible?

  1. Version 2 writes both formats; a later version drops the old one
  2. Version 2 converts every save and deletes the old format
  3. Version 2 ships without a rollback plan, to save time
Show the answer

Version 2 writes both formats; a later version drops the old one

While both formats are written, version 1 can still read every save. That is expand and contract.

Practise it

A rollback nobody has tried may fail on the day it is needed. Shard Garden's pipeline has a one-step rollback, and Priya runs it on staging every week.

The SRE book's advice for incidents applies: "Use the process routinely so it becomes second nature."

Quick check

Why does the phone app keep risky new features behind a switch the server controls?

  1. An installed app can't be rolled back, but a switch can be turned off in seconds
  2. Switches make the app smaller
  3. The stores require every feature to have one
Show the answer

An installed app can't be rolled back, but a switch can be turned off in seconds

Players who already have the update keep it. A switch on the server turns the feature off for them without a new release.

Switches need rules

Feature flags turn code on and off without a release. At Shard Garden's size, they need rules of their own.

Bolt says: Next: feature flags at scale.

Fact-checked 12 October 2026