Percentiles: p50, p95, p99

Play this lesson

The average looks fine

The dashboard says Shard Garden's average save takes 198 ms. Priya, who runs observability, looks at the 100 saves behind it: 98 took 100 ms, and 2 took 5,000 ms.

(98 × 100 + 2 × 5,000) ÷ 100 = 198. No save took anything like 198 ms. Most were faster, and two players waited five seconds.

An average mixes everyone together. Percentiles show who waited how long.

p50, p95, p99

A percentile sorts every measurement from fastest to slowest. The Speed stage used the 75th for field data. Teams watch three more:

Google's SRE book says a high percentile, like the 99th or 99.9th, shows a plausible worst case. Its example: a typical request takes about 50 ms, but 5% are 20 times slower.

Do the math

Priya sorts 1,000 saves from fastest to slowest. Roughly which save is the p99?

  1. The 990th
  2. The 500th
  3. The 999th
Show the answer

The 990th

99% of 1,000 is 990, so about 990 saves are at least that fast and 10 are slower. The 500th is the p50, the median.

Why averages lie

A footnote in Google's SRE book: if 1% of requests take 50 times the average, the rest are about twice as fast as the average. The average sits where almost no request is.

It can also stay still while things get worse. The book warns that alerting only on average latency can show no change over a whole day, while the slowest requests change a lot.

Quick check

The average save time is the best number to alert on, because it includes every save.

Show the answer

False

Including every save is the problem: a few very slow saves barely move it. The SRE book shows an average staying flat while the slow tail changes. Watch p95 or p99.

Rare for one, common for many

"Only 1 in 100" sounds rare. But one co-op session can make 200 saves. The chance that all 200 beat the p99 is 0.99 multiplied by itself 200 times: about 0.13.

So about 87% of those sessions meet at least one p99-slow save. The same happens when one tap calls many services: if each is slow 1% of the time, a join that calls 10 of them is slow about 10% of the time.

The tail is what busy players actually feel.

Read the numbers

save time in ms, two builds
         p50  p95   p99Build A   90  180   250Build B   80  160  4100

Which build should Priya worry about?

  1. Build B: 1 in 100 saves takes over 4 seconds
  2. Build A: its p50 is slower
  3. Neither: both p95s are under 200 ms
Show the answer

Build B: 1 in 100 saves takes over 4 seconds

Build B is faster for most saves, but its p99 shows a slow tail: 1 in 100 saves waits over 4 seconds. In a 200-save session, most players meet it.

Percentiles across many servers

Shard Garden runs 40 match servers. Prometheus's docs warn that averaging their percentiles gives statistically nonsensical values, and that a summary, which works out percentiles on each server, can't be combined across servers.

Histograms can: add up every server's buckets, then work out the percentile once:

histogram_quantile(0.95, sum by (le) (rate(save_duration_seconds_bucket[5m])))

The answer is only as precise as the buckets, so put bucket edges near your targets.

Quick check

Priya has the p95 from each of 40 match servers. How should the studio get one p95 for the whole game?

  1. Add up all the servers' histogram buckets, then compute the p95 once
  2. Average the 40 p95 values
  3. Take the largest of the 40 p95 values
Show the answer

Add up all the servers' histogram buckets, then compute the p95 once

Prometheus warns that averaging quantiles gives nonsensical values. Summed histogram buckets describe every request together, so one p95 computed from them is correct.

Which part was slow?

A p99 tells us how slow the slowest requests are. It can't tell us which part of the journey made them slow.

Bolt says: Next: find a percentile yourself, in 10 saves.

Fact-checked 11 October 2026