Dashboards that answer a question
64 panels and no answer
Kai built a dashboard with 64 panels: CPU for every server, memory, every counter in the game. During Saturday's slowdown, the team stared at it for ten minutes. Nobody could say whether players were hurting.
The docs of Grafana, a popular open-source dashboard tool, put it simply: a dashboard should tell a story or answer a question. It should reduce cognitive load, the effort it takes to understand, not add to it.
The four golden signals
- LatencyHow long requests take. Keep successful and failed requests apart.
- TrafficHow much demand there is, like joins per second.
- ErrorsRequests that fail, outright or by policy: if we promise one second, a slower answer counts as an error.
- SaturationHow full the service is, like garden slots in use.
From Google's SRE book: if you can measure only four things about a user-facing system, measure these.
Pick the signal
A match server reports that 92% of its garden slots are in use. Which golden signal is that?
- Saturation
- Traffic
- Latency
Show the answer
Saturation
Saturation is how full a service is. Traffic would be how many joins arrive each second, and latency how long each one takes.
RED for services, USE for machines
- REDRate, Errors, Duration: requests per second, how many fail, and how long they take. Created by Tom Wilkie in 2015, for every service.
- USEUtilization, Saturation, Errors, from Brendan Gregg: for every resource, like a CPU or a disk.
Wilkie's summary: RED is about caring how happy your users are, USE about how happy your machines are.
Symptoms first, causes below
The SRE book says monitoring should answer two questions: what's broken, and why. "What's broken" is the symptom. The "why" is a cause.
Priya, who runs observability, redraws the dashboard. The top row shows symptoms players feel: join success, save time at p95, crashes. Below, one row per service with its RED numbers. At the bottom, machines with USE. Every graph marks each release, so "it started with 1.4.2" is visible at a glance.
Quick check
If failed joins return an error in 5 ms, mixing them into the join latency graph can make latency look better while things get worse.
- True
- False
Show the answer
True
Fast failures drag the numbers down. The SRE book says to keep the latency of successful and failed requests apart, and adds that a slow error is even worse than a fast one.
Read the numbers
1 now usual2Join success 97.1% 99.9%3Join p95 2.9 s 0.6 s4Match CPU 41%5DB connections 498 of 500
Join success and join p95 are the symptoms. Which row points to the likely cause?
- Garden DB connections: 498 of 500 in use
- Match server CPU at 41%
- None: the symptoms are enough
Show the answer
Garden DB connections: 498 of 500 in use
The database has almost no free connections, so new joins queue and time out: saturation. CPU at 41% has plenty of room.
Fewer, better dashboards
Grafana warns about dashboard sprawl: dashboards multiplying until nobody knows which one is right. Priya's rules for the studio:
- Each dashboard has one question as its title.
- The same time range on every panel.
- Units on every axis.
- A dashboard nobody opened in three months gets deleted.
Quick check
Which panel belongs in the top row of Priya's main dashboard?
- Share of garden joins that succeed
- CPU of match server 17
- Bytes written by the log shipper
Show the answer
Share of garden joins that succeed
The top row shows what players feel, and failed joins are a symptom they notice at once. The others are causes or internals, useful further down while digging.
From a graph to a promise
The top row says how players are doing right now. The next step is deciding how good is good enough, and writing it down.
Bolt says: Next: SLIs and SLOs, the studio's promises in numbers.
Fact-checked 11 October 2026
- Google SRE book: Monitoring distributed systems sre.google
- Grafana: Dashboard best practices grafana.com
- Grafana Labs: The RED method (2018) grafana.com
- Brendan Gregg: The USE method brendangregg.com