Chaos engineering
Break it on purpose
Shard Garden is built to survive a server failing. Nobody knows if that is true until a server actually fails.
The Principles of Chaos Engineering define chaos engineering as "the discipline of experimenting on a system in order to build confidence in the system's capability to withstand turbulent conditions in production."
An experiment in four steps
The principles describe each experiment:
- Steady state"Some measurable output of a system that indicates normal behavior."
- HypothesisThe steady state "will continue in both the control group and the experimental group."
- Real-world eventsIntroduce things like "servers that crash".
- Try to disprove itLook "for a difference in steady state".
Quick check
Which would make a good steady state for Shard Garden?
- Weed races started per minute
- No code changes for a week
- Every server at 0% CPU
Show the answer
Weed races started per minute
A steady state is a measurable output that shows normal behaviour, something players would feel if it changed.
Chaos Monkey
Netflix built a tool called Chaos Monkey. It "is responsible for randomly terminating instances in production", so that "engineers implement their services to be resilient to instance failures."
A team that knows servers vanish at random designs for it every day.
Write the hypothesis
Priya will stop one of the three race servers. Which is a real hypothesis?
- "Races started per minute stay the same when one server stops"
- "A server will stop"
- "Nothing will ever go wrong"
Show the answer
"Races started per minute stay the same when one server stops"
A hypothesis names the steady state and predicts it holds. The experiment then tries to prove it wrong.
Keep the blast small
One principle is "Minimize Blast Radius": "Experimenting in production has the potential to cause unnecessary customer pain."
Shard Garden starts each experiment on staging, then on a small share of production, during working hours, with Priya watching and a way to stop it at once.
Quick check
The studio's first chaos experiment should switch off a whole region in production.
- True
- False
Show the answer
False
Start small and contained. A whole region is the biggest possible blast, saved for when smaller experiments pass.
A failed experiment is a win
If the steady state changes, the team has found a weakness before players did. They fix it, then run the experiment again. The principles go further: "Automate Experiments to Run Continuously".
Bolt says: Next: having enough servers before players need them.
Fact-checked 12 October 2026
- Principles of Chaos Engineering principlesofchaos.org
- Netflix: Chaos Monkey netflix.github.io