What observability is
Twenty seconds of nothing
Shard Garden is live: thousands of co-op gardens run on our match servers at once. On Friday night a player posts: "Our garden froze for 20 seconds, then all four of us got kicked."
Kai checks one server: fine. Remy plays a match: fine. Nobody can see what happened inside that one garden.
That week Priya joins the studio. Priya runs observability: the tools and habits that let the team see inside Shard Garden while players are in it.
Asking new questions
OpenTelemetry, an open-source project for collecting this kind of data, says observability lets you understand a system from the outside by asking it questions. The data a system sends out about its own behaviour is telemetry.
Monitoring watches for problems someone predicted: "is the server up?" Observability also helps with unknown unknowns, problems nobody predicted. Priya can ask "why did garden 7f3a freeze?", a question nobody built a graph for.
Quick check
Observability means the team can only answer the questions it built dashboards for in advance.
- True
- False
Show the answer
False
That is the limit of plain monitoring. Observability is about asking new questions, like why one garden froze, from the telemetry the game already sends.
Three main signals
- LogsTimestamped messages from a service: "garden 7f3a: save failed, retrying".
- MetricsMeasurements taken while the game runs, like the number of live gardens every 15 seconds.
- TracesThe path one request takes, for example a "Join garden" tap travelling through four services.
OpenTelemetry calls each kind of telemetry a signal. It also lists baggage, extra context passed between signals, and is still working on two more: events and profiles.
Pick the signal
Remy wants to know which services one slow "Join garden" request passed through, and where it waited. Which signal answers that?
- A trace
- A metric
- A single log line
Show the answer
A trace
A trace records the path of one request across every service, with the time spent in each. A metric gives totals over many requests, and one log line shows one moment in one service.
From inside and from outside
Google's SRE book, written by its site reliability engineers, names two kinds of monitoring:
- White-box: numbers the system reports about its own insides, like a match server's queue length.
- Black-box: testing what users can see, the way a user would.
Priya sets up a probe, a small robot player that joins a test garden every minute from three regions. If it can't join, players probably can't either, whatever the insides say.
Quick check
Which of these is black-box monitoring?
- A robot player that joins a test garden every minute
- The match server reporting its memory use
- The save service counting its database errors
Show the answer
A robot player that joins a test garden every minute
Black-box monitoring tests what a player would see from the outside. The other two are white-box: numbers the system reports about its own insides.
True story: Roblox's 73-hour outage
Starting on 28 October 2021, Roblox, used by fifty million players every day, was down for 73 hours. The cause was inside Consul, the system Roblox used for service discovery: how its servers find each other. A new streaming feature and a problem in BoltDB, a database inside Consul, combined.
Daniel Sturman and colleagues from Roblox and HashiCorp wrote why it took so long: the monitoring that would have shown the cause also relied on Consul. Roblox removed that circular dependency.
Telemetry has to survive the failure it should explain.
Source: Roblox: Roblox return to service 10/28-10/31 2021, about.roblox.com
Plan it
Where should our monitoring run? The match servers have spare room, so I could put it there.
What should the studio do?
- Run it apart from the match servers, so it still works when they fail
- Put it on the match servers to save money
- Skip monitoring and read Discord instead
Show the answer
Run it apart from the match servers, so it still works when they fail
Monitoring that runs on the system it watches goes dark at the moment it is needed most, as Roblox found. Keep it independent.
Start with single events
The oldest signal is the log. Most games already write logs, but usually as sentences only a person can read.
Bolt says: Next: structured logs, which a computer can search in a second.
Fact-checked 11 October 2026
- OpenTelemetry: Observability primer opentelemetry.io
- OpenTelemetry: Signals opentelemetry.io
- Google SRE book: Monitoring distributed systems sre.google
- Roblox: Roblox return to service 10/28-10/31 2021 about.roblox.com