On-call and runbooks

Play this lesson

Who answers at 3 a.m.?

An incident is anything that stops players playing or puts their data at risk. Shard Garden has players in every time zone, so incidents don't wait for office hours.

Priya, who runs observability, sets up an on-call rotation. Each week one teammate is primary, paged first, and another is secondary, the backup. Kai and Remy take daytime shifts. The SRE book gives typical response times: 5 minutes for user-facing services, 30 for less urgent ones.

Keeping on-call humane

Google's SRE book sets limits so on-call stays sustainable:

It also stresses keeping stress low. Under pressure, people tend to act fast and on instinct, when a complex problem needs calm, deliberate thinking.

Quick check

Google's SRE book plans for at most 2 incidents per 12-hour on-call shift.

Show the answer

True

Each incident takes time to fix, find the root cause and follow up. More than two a shift means overload, and the alerts or the system need work.

Runbooks

A runbook, also called a playbook, is a written guide for one alert: what it means, how to check what is happening, how to fix it, and who to call next.

The SRE book says recording best practices ahead of time in a playbook gives roughly a 3x improvement in MTTR, the mean time to repair, compared with "winging it". Every page Priya creates links straight to its runbook.

Runbook pick

Which is the best first step in the runbook for "Garden DB connections near limit"?

  1. Open the database dashboard and check which service holds the most connections
  2. Restart every match server at once
  3. Ask in Discord whether players notice anything
Show the answer

Open the database dashboard and check which service holds the most connections

A runbook checks before it acts. Restarting everything could kick every player out of their garden, and Discord is not a measurement.

Runbooks need a safe home

A runbook only helps if the on-call person can open it during the outage. So Priya keeps a copy outside the game's own systems, and every runbook says how to reach the servers if the normal way is down.

Runbooks go stale, so each one has an owner and a date, and the on-call person fixes anything wrong right after the incident.

True story: locked out of their own buildings

On 4 October 2021, Facebook's apps went offline around the world. Santosh Janardhan of Facebook explained: a command meant to check the backbone network's capacity took it down, and a bug in an audit tool failed to stop it. Its DNS servers became unreachable, so the internet couldn't find Facebook's servers.

Engineers couldn't reach the data centres remotely: those networks were down. They went on site, where strong security slowed them. Facebook said its "storm" drills helped bring things back.

Rehearse the hard way back before you need it.

Source: Engineering at Meta: More details about the October 4 outage, engineering.fb.com

Plan it

Priya, #on-call
Our runbooks live on a wiki that runs on the same servers as Shard Garden.

What is the risk?

  1. In a big outage the runbooks go down with the game, exactly when they are needed
  2. Wikis are too slow for runbooks
  3. None, because the servers are reliable
Show the answer

In a big outage the runbooks go down with the game, exactly when they are needed

Like Facebook's engineers locked out by their own network, a runbook on the failing system is out of reach when it matters. Keep copies somewhere independent.

Practise on a calm day

Facebook's storm drills take a service, a data centre or a whole region offline on purpose. Smaller teams run game days: on a planned morning, Remy safely breaks something in the test environment, a test page fires, and the on-call person follows the runbook while everyone watches.

Afterwards, the team fixes what slowed them down. Every real page gets the same review: was it urgent, was it actionable, did the runbook work?

Quick check

Priya was paged six times last night for the same alert, and each time the fix was "restart the save worker". What should change?

  1. Automate the restart and find the root cause, so the alert stops paging people
  2. Hand the pager to someone else
  3. Turn the phone's volume down at night
Show the answer

Automate the restart and find the root cause, so the alert stops paging people

The SRE book calls a page with a rote answer a red flag: a machine can do the restart. Six pages a night is also far past the two-incidents-per-shift limit.

When the game itself falls over

Some incidents start on a single phone: a crash. To fix it, the team first has to read what the crash report says.

Bolt says: Next: crash reports and symbolication.

Fact-checked 11 October 2026