Log levels, and what never to log

Play this lesson

40 GB of "tick ok"

Priya, who runs observability, looks at the bill. Shard Garden's match servers write 40 GB of logs a day. Most of it is one line, tick ok, written 30 times a second in every garden.

Somewhere inside is the line that matters: a save that failed. Every log costs money to send and store, and noise hides the signal. Log levels help: each event gets a level that says how important it is.

The levels

These are OpenTelemetry's names. Each also has a number range, from TRACE at 1 to 4 up to FATAL at 21 to 24, so tools can compare levels from any programming language.

Pick the level

The match server can't reach the save service, retries, and the save works on the second try. Which level fits?

  1. WARN
  2. ERROR
  3. FATAL
Show the answer

WARN

Nothing failed for the player, but it matters more than routine information: if retries pile up, something is wrong. ERROR is for things that actually went wrong, and FATAL for crashes.

A level for each place

Each service has a minimum level and drops everything below it. On Kai's laptop the game logs DEBUG and up. In production the match servers log INFO and up, so tick ok, now a DEBUG line, never leaves the server.

When Priya hunts a bug, the studio switches one service to DEBUG for 30 minutes with a config flag, a setting changed without a new release. Then it goes back.

Quick check

Shard Garden's match servers should log at TRACE in production all the time, so nothing is ever missed.

Show the answer

False

TRACE everywhere multiplies the cost and buries the important lines. Production logs INFO and up, and DEBUG is switched on briefly, for one service, while hunting a bug.

What never goes in a log

Logs are copied to many places and read by many people. OWASP lists data that should usually not be recorded directly, but removed, masked, hashed or encrypted:

For a game played by children, the studio adds chat messages, email addresses and real names.

True story: passwords in readable form

On 21 March 2019, Pedro Canahuati, Facebook's VP of Engineering, Security and Privacy, wrote that a routine security review in January had found some user passwords stored in a readable format in internal data storage systems.

Facebook planned to notify hundreds of millions of Facebook Lite users, tens of millions of Facebook users and tens of thousands of Instagram users. In April it found more logs of Instagram passwords: millions of users. The passwords were never visible outside Facebook.

A secret can leak into storage nobody checks.

Source: Meta: Keeping passwords secure (2019), about.fb.com

Bug hunt

login.js
async function login(req) {  log.info({    event: 'login.attempt',    player_id: req.body.id,    body: req.body });  const ok = await    verify(req.body);  log.info({ ok });}

Which line of Remy's code could put players' passwords in the logs?

  1. Line 4
  2. Line 5
  3. Line 8
Show the answer

Line 5

Line 5 logs the whole request body, and the body holds the password the player typed. Log the event and the player ID, never the whole request.

Log the fact, not the secret

Some events should always be logged. OWASP lists input validation failures, login successes and failures, access control failures, and session problems, like a tampered session cookie.

Log that it happened, never the secret involved:

{"event":"login.failed", "player_id":"p-91c2", "reason":"wrong_password"}

Now Priya can spot 500 failed logins on one account in a minute, which looks like someone guessing, without a single password in the log.

Quick check

Which log line is safe to keep?

  1. {"event":"login.failed", "player_id":"p-91c2", "reason":"wrong_password"}
  2. {"event":"login.failed", "player_id":"p-91c2", "typed":"sprig2015"}
  3. {"event":"token.issued", "player_id":"p-91c2", "token":"eyJhbGciOi..."}
Show the answer

{"event":"login.failed", "player_id":"p-91c2", "reason":"wrong_password"}

The first records the fact and an internal ID. The second keeps a typed password and the third an access token, both on OWASP's list of things to keep out of logs.

Counting instead of telling

Logs tell the story of single events. Many questions are about totals: how many gardens, how many saves, how fast.

Bolt says: Next: metrics, which count everything for a fraction of the cost.

Fact-checked 11 October 2026