Incident command

Everyone fixing at once

Weed races stop starting. Remy and five other teammates jump in. Two restart the same server at once, one changes a setting nobody else knows about, and nobody tells the players.

Google's SRE book has a chapter on running an incident so this doesn't happen.

Four roles

Each person in an incident takes one role:

• Incident command"Holds the high-level state about the incident" and hands out the work.
• Operations"The only group modifying the system during an incident."
• Communication"The public face of the incident response task force."
• PlanningSupports operations "by dealing with longer-term issues".

Is it an incident?

The book says to declare an incident if any one of these is true:

"Do you need to involve a second team in fixing the problem?" "Is the outage visible to customers?" "Is the issue unsolved even after an hour's concentrated analysis?"

A living document

"The incident commander's most important responsibility is to keep a living incident document": what is known, who is doing what, and a timeline of every change.

Anyone who joins late reads it instead of interrupting the people fixing things.

Handing over

Incidents can last for hours, and people get tired. The book gives the words for a handover: "You're now the incident commander, okay?"

The old commander "should not leave the call until receiving firm acknowledgment of handoff."

Practise when it's calm

The book's advice: "Use the process routinely so it becomes second nature", and "Give full autonomy within the assigned role to all incident participants."

Shard Garden runs a practice incident on staging every month, with different people in each role.

Bolt says: Next: telling players what is happening.