Blameless postmortems
After the fire
Google's SRE book defines a postmortem as "a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions to prevent the incident from recurring."
When to write one
The book lists common reasons to write a postmortem:
- Players noticedDowntime or slowness "beyond a certain threshold".
- Data lost"Data loss of any kind".
- On-call actedA person stepped in, such as a "release rollback".
- Monitoring missed itSomeone found it by hand, not an alert.
Quick check
A release was rolled back after 4 minutes, and no player noticed. Does it need a postmortem?
- Yes: an on-call rollback is one of the reasons
- No: nobody noticed
- Only if data was lost
Show the answer
Yes: an on-call rollback is one of the reasons
A rollback means something reached production that shouldn't have. That is worth understanding while it is still small.
Blameless, and why
A postmortem must "focus on identifying the contributing causes of the incident without indicting any individual or team".
The book explains: "You can't 'fix' people, but you can fix systems and processes to better support people making the right choices." People who expect blame hide mistakes, and hidden mistakes teach nobody.
True story: 27 minutes at Cloudflare
On 2 July 2019, Cloudflare, the network the studio's own website runs on, deployed new firewall rules. One regular expression in them "backtracked enormously", pushing CPUs to nearly 100% on servers around the world. Cloudflare's service went down for 27 minutes.
The postmortem found that everything before the deploy was done "correctly", and warned: "Getting to a single root cause, while satisfying, may obscure the reality." Its fixes included rolling out rules in stages, like Cloudflare's other software.
Source: Cloudflare: Details of the Cloudflare outage on July 2, 2019, blog.cloudflare.com
Quick check
Cloudflare's postmortem blamed no one. What did it change instead?
- How rules are released: in stages, like its other software
- Who is allowed to write rules
- Nothing
Show the answer
How rules are released: in stages, like its other software
Rules had gone out everywhere at once, by design, to stop attacks fast. Staged rollouts change the system, so the next bad rule hits a few servers first.
Actions with owners
A postmortem ends with action items. At Shard Garden, Priya checks that each one names a single owner and a date, and goes into the team's tracker. An action that belongs to everyone belongs to no one.
Fix the action item
Which action item belongs in Shard Garden's postmortem?
- "Remy: add a CPU-time test for every firewall rule, by 30 October"
- "Be more careful with rules"
- "Kai should not deploy on Fridays"
Show the answer
"Remy: add a CPU-time test for every firewall rule, by 30 October"
It is specific, owned and dated, and it changes the system. The others are vague or blame a person.
The worst kind of incident
Some incidents lose data. How much a studio can afford to lose is a decision made before it happens.
Bolt says: Next: backups, RPO and RTO.
Fact-checked 12 October 2026
- Google SRE book: Postmortem culture sre.google
- Cloudflare: Details of the Cloudflare outage on July 2, 2019 blog.cloudflare.com