Sampling

Play this lesson

Too many traces to keep

On a busy evening, Shard Garden makes about 5,000 traces a second, each with dozens of spans. Storing them all would cost more than the match servers themselves.

Sampling means keeping only some of them. OpenTelemetry calls it one of the most effective ways to cut the cost of observability without losing visibility. The trick is choosing which ones to keep.

Why a sample works

A small sample can stand for a large group, the way a survey of 1,000 people can describe a whole country. OpenTelemetry's guide says that for systems with lots of traffic, a sampling rate of 1% or lower can very accurately represent the other 99%.

Its rule of thumb: consider sampling from about 1,000 traces a second. With only tens of small traces a second or fewer, keep everything.

Quick check

The studio's website makes about 20 small traces a minute. It should sample them at 1%.

Show the answer

False

That is far below the point where sampling helps. OpenTelemetry advises against it for tens of small traces a second or fewer, and at 1% the site would keep one trace every five minutes.

Head sampling

Head sampling decides as early as possible, when the trace starts. The most common kind works from the trace ID and a percentage, for example 5%. Every service makes the decision from the same trace ID, so they all agree: a trace is kept whole or not at all.

The catch: the decision comes before anything has happened. Head sampling can't make sure every trace with an error is kept, because the error hasn't happened yet.

Do the math

Shard Garden makes 5,000 traces a second, and 1 in 1,000 has an error. With 2% head sampling, about how many error traces does Priya keep each second?

  1. 0.1
  2. 5
  3. 100
Show the answer

0.1

5,000 traces include 5 with errors. Head sampling keeps 2% of everything, errors included: 0.1 a second, one every 10 seconds. 100 is all the kept traces, mostly boring ones.

Tail sampling

Tail sampling decides after seeing all or most of a trace's spans. Then it can keep exactly what matters:

OpenTelemetry warns that it is hard to build and to run, because it must store a lot of data until each trace ends. The OpenTelemetry Collector has a tail sampling processor, and head and tail sampling can be combined.

Pick the sampler

Priya, #observability
I want every trace where a join failed or took over 5 seconds, plus 1% of the normal ones.

Which kind of sampling can do that?

  1. Tail sampling, because it sees the whole trace first
  2. Head sampling at 1%
  3. Head sampling at 100%
Show the answer

Tail sampling, because it sees the whole trace first

Only tail sampling knows how long the join took and whether it failed, because it decides after the trace ends. Head sampling decides at the start, and 100% means keeping everything.

Sample traces, count everything

Metrics don't need sampling: a counter adds 1 for every save, whatever happens to the traces. So Priya counts errors exactly with metrics, and uses traces as detailed examples.

If you ever count sampled traces, scale back up. At 2%, each kept trace stands for 50. That only works if you know the sampling rate each trace was kept at.

Quick check

With 2% head sampling, each kept trace stands for about 50 traces.

Show the answer

True

2% is 1 in 50, so each kept trace stands for itself and 49 that were dropped. Counts from metrics don't need this, because they include every event.

Putting it on one screen

Logs, metrics and sampled traces are all flowing. Now the team needs one screen that says, at a glance, whether players are OK.

Bolt says: Next: dashboards that answer a question.

Fact-checked 11 October 2026