When things break: outages, attacks and backups

Play this lesson

Saturday, 10 a.m.: nothing loads

The Halloween event starts, and Discord fills up: "Hyper Shards won't load!" "My garden is back at wave 1!" This is an incident: anything that stops players playing or puts their data at risk. Every live game has them. Studios that recover fast aren't luckier. They planned on a calm day.

DDoS: a traffic jam on purpose

A denial-of-service attack floods a server with so much traffic that it can't answer real players. In a distributed one, a DDoS, many machines attack together, often a botnet: hijacked computers and gadgets whose owners have no idea.

One small server can't outmuscle that. A CDN like Cloudflare can, thanks to anycast: its data centres all answer at the same address, so the flood is spread over the whole network instead of piling onto one building. Cloudflare includes automatic, unmetered DDoS protection on every plan, even the free one, for traffic that flows through it.

Quick check

Why can Cloudflare handle a DDoS that would knock out one rented server?

  1. Its data centres all answer at the same address, so the flood is spread across the whole network
  2. It asks the attackers' internet providers to switch them off first
  3. It turns the website off until the attack is over
Show the answer

Its data centres all answer at the same address, so the flood is spread across the whole network

With anycast, the attack lands on many data centres at once instead of one machine. Each one handles a slice, and Cloudflare's automatic systems filter the bad traffic out.

Backups: the 3-2-1 rule

A backup is a copy we can bring back after a disaster. CISA, the US cyber defence agency, recommends the 3-2-1 rule.

For saves, Firestore offers scheduled backups, daily or weekly and kept for up to 14 weeks, on the Blaze plan. A restore goes into a new database, never over the live one. Point-in-time recovery, off until we switch it on, keeps one version per minute for the last 7 days, for the day someone deletes the wrong thing.

Ransomware and the restore test

Ransomware is malicious software that scrambles (encrypts) files so their owner can't open them, then demands money to unscramble them. CISA warns that many kinds hunt for backups they can reach and delete or scramble those too. So it advises offline, encrypted backups, like a drive kept unplugged by a parent, and regular tests.

The test matters most. A backup only counts once you have restored it and watched the game work. Kai and Remy practise monthly: restore last week's backup into a test database and load a save from it.

Backup check

#studio-chat, Remy
Our backup plan: Firestore's daily backup. It has run every night for a year, so we're totally safe, right?

What is the most important thing missing from Remy's plan?

  1. A test restore, plus a second copy kept offline somewhere else
  2. Nothing, a year of nightly backups proves they work
  3. Backups every minute instead of every night
Show the answer

A test restore, plus a second copy kept offline somewhere else

A backup that has never been restored might not work, and one copy in one place breaks the 3-2-1 rule. Anyone who took over the Firebase account could delete those backups too, but not a drive in a drawer.

The incident plan

An incident plan is written on a calm day, so nobody has to invent it in a panic.

True story: the PlayStation Network goes dark

Between 17 and 19 April 2011, attackers broke into Sony's PlayStation Network, the online service for PlayStation games. On 20 April, Sony switched it off. The attackers may have taken names, addresses, email addresses, birthdates and passwords, and Sony emailed all 77 million registered accounts.

The network began coming back on 15 May, country by country, with more encryption, extra firewalls and password resets. Sony offered a "Welcome Back" package, including free downloads and 30 days of PlayStation Plus.

Stop the damage, change the keys, tell players: the same incident plan, for 77 million accounts.

Source: PlayStation Blog: update on PlayStation Network and Qriocity (26 April 2011), blog.playstation.com

Status update

Hyper Shards stopped loading 20 minutes ago after the morning update. Which post should go in our Discord #announcements?

  1. "Hyper Shards isn't loading since 10:05. We're rolling back this morning's update now. Next update here by 10:45, even if nothing has changed."
  2. "Some technical difficulties. Stay tuned!"
  3. No post until it's fixed, so players don't panic.
Show the answer

"Hyper Shards isn't loading since 10:05. We're rolling back this morning's update now. Next update here by 10:45, even if nothing has changed."

A good update says what happened, what we're doing and when the next update comes. A vague post gives players nothing, and silence leaves them guessing and posting angry reviews.

The blameless post-mortem

When the fire is out, the team writes a post-mortem: what happened, how it hurt players, what we did, the root causes, and follow-up actions. Google's site reliability engineers, who keep some of the busiest services on Earth running, make it blameless: it looks for causes in systems and processes, not a person to punish. Their reason: if people fear blame, they hide problems.

So not "Kai shipped a bug", but "no test covered old saves, so a bad update reached everyone. Actions: add the test, use staged rollouts."

Let finders reach you: security.txt

Sometimes a stranger spots a weakness in our game and wants to tell us. Vulnerability disclosure (often called responsible disclosure) means reporting it privately, so we can fix it before anyone abuses it. RFC 9116, an internet standard from 2022, gives finders a place to look: a file at /.well-known/security.txt, served over HTTPS.

Contact: mailto:security@ourstudio.com
Expires: 2027-10-01T00:00:00.000Z
Acknowledgments: https://ourstudio.com/thanks

Contact and Expires are required, and Expires should be less than a year away. The UK's NCSC advises: reply promptly, and thank them.

Bolt says: Plan on a calm day, back up three ways, practise the restore, and fix systems, not people. Now take the boss quiz.

Fact-checked 11 October 2026