Error budgets

Play this lesson

Ship it, or hold it?

Kai wants to ship the big level editor update on Friday. Priya, who runs observability, says joins have failed a lot this month. Kai says players are waiting for the editor. Both have a point, and arguing won't settle it.

An error budget settles it with a number agreed in advance: the amount of failure the SLO allows. While budget is left, ship. When it is spent, fix reliability first.

How big is the budget?

The budget is 100% minus the SLO. Google's SRE book example: an SLO of 99.999% of queries per quarter leaves a budget of 0.001%. A problem that fails 0.0002% of the quarter's queries spends 20% of it.

Every failure counts, whatever caused it: our bugs, a bad release, even a data centre outage.

For Shard Garden: a 99.9% join SLO over four weeks with 3 million joins leaves 3,000 failed joins to spend.

Do the math

Shard Garden's budget is 3,000 failed joins this window. Remy's bad matchmaker release failed 750 joins. How much of the budget did it spend?

  1. 25%
  2. 7.5%
  3. 75%
Show the answer

25%

750 of 3,000 is a quarter, 25%. Under Google's example policy, later in this lesson, one incident that spends over 20% of the budget gets a postmortem.

A budget is for spending

The SRE book says users typically don't notice the difference between high and extreme reliability: someone on a 99% reliable phone can't tell 99.99% from 99.999%.

So budget left over is not a prize. It can pay for faster releases, experiments, and risky but useful work, like moving the save database. A team that never spends its budget is probably being slower than it needs to be.

Quick check

Shard Garden should aim for 100% join success, because more reliability is always better.

Show the answer

False

The SRE book says 100% is probably never the right target. Players can't notice the last fractions of a percent, and chasing them would stop the studio shipping anything.

The error budget policy

Google's SRE workbook has an example error budget policy, agreed before anything goes wrong:

It is not a punishment: it lets the team focus on reliability.

Policy call

Kai, #releases
We are 15% over our join budget for the last four weeks, but the editor update is ready. Ship it Friday?

Under the example policy, what happens?

  1. Hold the editor update and work on join reliability until the service is back within its SLO
  2. Ship it, because the editor isn't part of joining
  3. Ship it, and lower the SLO afterwards
Show the answer

Hold the editor update and work on join reliability until the service is back within its SLO

Over budget means all changes and releases stop except urgent and security fixes. A new release is risk, and the budget for risk is gone.

Agreed on a calm day

The policy works because both sides sign it before it is needed. Kai and Remy, who decide what to build, and Priya, who watches reliability, agree it in a quiet week.

There are gentler steps before a full stop: the SRE book mentions slowing releases down or rolling them back as the budget runs low. And a team that keeps struggling to ship may loosen its SLO to get a bigger budget.

Quick check

During a release freeze under the example policy, a security fix can still ship.

Show the answer

True

The example policy halts all changes except P0 issues and security fixes. Leaving a security hole open would put players at more risk than the fix does.

Who notices the budget burning?

A budget only helps if someone notices it running out, ideally long before it is gone, and without watching graphs all night.

Bolt says: Next: alerts that fire for the right reasons.

Fact-checked 11 October 2026