Metrics: counters, gauges, histograms

How many gardens, right now?

Priya, who runs observability, wants a graph of live gardens, every 15 seconds, for the last month. Counting garden.start log lines would mean reading billions of them.

A metric is a measurement of a service taken while it runs. Instead of one line per event, the server keeps a running number and reports it every few seconds, so the cost hardly changes whether 10 or 10,000 gardens are playing.

Four metric types

• CounterOnly goes up, or back to zero when the server restarts. Saves done, errors.
• GaugeGoes up and down. Gardens live right now, memory in use.
• HistogramCounts measurements in buckets, like join times up to 0.1 s, up to 0.5 s, and so on.
• SummaryWorks out percentiles on the server itself.

These are the names in Prometheus, an open-source monitoring system. OpenTelemetry has the same ideas, plus an UpDownCounter: a count that can also go down, like players in a lobby.

Counters become rates

garden_saves_total only grows: 1,204,511 saves since the server started. That number alone says little. Dashboards turn it into a rate: how fast it grew over the last 5 minutes, for example 420 saves a second.

Prometheus's rule: do not use a counter for a value that can decrease. Players in a lobby join and leave, so a counter would be wrong for them.

Inside a histogram

A histogram of join times sends three kinds of numbers:

• match_join_duration_seconds_bucket{le="0.5"}: joins that took 0.5 seconds or less. le means "less than or equal", so each bucket includes the smaller ones.
• ..._sum: all join times added together.
• ..._count: how many joins there were.

Sum divided by count gives the average. The buckets give much more: the shape of the whole spread, slow tail included.

Names and labels

Prometheus's naming rules: one unit per metric, in base units (seconds and bytes, not milliseconds and megabytes), with the unit at the end of the name, and _total on counters.

Labels split a metric: region="eu", build="1.4.2". Every combination of label values is a separate time series to store. So Prometheus warns: no labels with many possible values, like user IDs or email addresses.

The average is hiding something

Shard Garden's average save time looks great. Players still say saves sometimes hang. Both can be true.

Bolt says: Next: percentiles, p50, p95 and p99, and why averages lie.