Alerts that matter
40 buzzes a night
In Priya's first week, the phone of whoever was responsible that night buzzed 40 times: "CPU over 80% on match-eu-3". Every time, players were fine.
After a week of that, people stop reading alerts. That is alert fatigue, and it is how a real problem gets missed. Google's SRE book notes that a person can only react with a sense of urgency a few times a day before getting tired.
What deserves a page
A page is an alert that calls a person right away, even at night. The SRE book asks of every page: is it urgent, actionable and visible to users, now or soon?
- Every page should be actionable.
- Every page should need a person's judgement. One with a rote, step-by-step answer is a red flag: automate it.
- Page on symptoms players feel. Causes, like CPU, help with debugging.
Everything else becomes a ticket, fixed in working hours.
Page or ticket
Which of these should wake Remy at 3 a.m.?
- Join success has been under 95% for 10 minutes
- CPU on one match server is above 80%
- The log disk will be full in three weeks
Show the answer
Join success has been under 95% for 10 minutes
Failing joins are urgent, and players feel them now. High CPU with happy players is not a symptom, and the disk can be a ticket for working hours.
Four ways to judge an alert
- PrecisionOf the alerts that fired, how many were about something significant.
- RecallOf the significant events, how many caused an alert.
- Detection timeHow long it takes to send the alert.
- Reset timeHow long it keeps firing after the problem is fixed.
These come from Google's SRE workbook. The CPU alert had terrible precision.
Burn rate
The workbook alerts on the error budget, the share of requests an SLO allows to fail. Burn rate is how fast the service spends that budget compared with the SLO. With a 99.9% SLO over 30 days:
- Burn rate 1: 0.1% of requests fail, and the budget lasts exactly 30 days.
- Burn rate 10: 1% fail, gone in 3 days.
- Burn rate 1,000: everything fails, gone in 43 minutes.
It is the error rate divided by the error rate the SLO allows.
Do the math
Shard Garden's join SLO is 99.9%. Right now 2% of joins fail. What is the burn rate?
- 20
- 2
- 200
Show the answer
20
The SLO allows 0.1% to fail. 2% divided by 0.1% is 20, so a 30-day budget would last a day and a half.
Two windows
The workbook's recommended starting point:
- Page if 2% of the 30-day budget burns in 1 hour (burn rate 14.4).
- Page if 5% burns in 6 hours (burn rate 6).
- Ticket if 10% burns in 3 days (burn rate 1).
Each rule also checks a short window, 1/12 as long: 5 minutes for the 1-hour rule. Both must be over the limit, so the alert stops soon after a fix instead of firing for another hour.
True story: the warnings nobody received
On 31 January 2017, a GitLab engineer meant to remove data from a secondary database, but the command ran on the primary. GitLab lost about six hours of data: at least 5,000 projects, 5,000 comments and roughly 700 users.
The backups had been failing too. Each failure sent an email, but the emails were rejected by the receiving server. GitLab wrote that it was never aware of the backups failing until it was too late. Afterwards it added Prometheus monitoring for backups.
An alert path needs testing too.
Source: GitLab: Postmortem of database outage of January 31 (2017), about.gitlab.com
Quick check
If an alert hasn't fired for months, the thing it watches must be healthy.
- True
- False
Show the answer
False
Silence can also mean the alert path is broken, as GitLab found. Teams test their alerts on purpose, and some send a regular "still alive" signal, with an alert for when it stops.
Someone has to answer
A good page reaches the right person with a clear problem. Next: who that person is, and what they have in hand when the phone buzzes.
Bolt says: Next: on-call rotations and runbooks.
Fact-checked 11 October 2026
- Google SRE workbook: Alerting on SLOs sre.google
- Google SRE book: Monitoring distributed systems sre.google
- GitLab: Postmortem of database outage of January 31 (2017) about.gitlab.com