Feature flags at scale

Play this lesson

Forty switches

A feature flag, or toggle, chooses which code runs while the game is running. Shard Garden now has forty of them, and Remy is not sure what some of them still do.

Four kinds of flag

Pete Hodgson's guide on Martin Fowler's site sorts them by job, and by how long each should live:

Quick check

Priya adds a switch that turns weed races off when the servers struggle. Which kind is it?

  1. An ops flag (a kill switch)
  2. A release flag
  3. An experiment flag
Show the answer

An ops flag (a kill switch)

It controls how the system runs under load, not who sees a feature or which test group a player is in.

Every flag has a cost

Hodgson writes: "Savvy teams view the Feature Toggles in their codebase as inventory which comes with a carrying cost". Each flag adds another path through the code to test.

His fixes: add a removal task the day a flag is added, give flags expiry dates, even "time bombs" that fail a test once a flag is past its date, and cap how many flags exist.

True story: the flag that woke old code

On 1 August 2012, Knight Capital, a US share-trading firm, put new code on its eight order servers. One technician did not copy it to the eighth server, and no second person checked.

The new code reused an old flag. On seven servers, the flag turned on the new feature. On the eighth, it woke "Power Peg", code unused for years but never deleted. In about 45 minutes, the system made over 4 million trades, and Knight lost $460 million.

Source: SEC: Order against Knight Capital Americas LLC (16 October 2013), sec.gov

Find the habit

Which habit would have left the eighth server nothing to wake?

  1. Delete unused code, and give every new feature a brand-new flag
  2. Deploy on fewer servers
  3. Deploy only on Mondays
Show the answer

Delete unused code, and give every new feature a brand-new flag

The old flag still meant "run Power Peg" to the old code. A new flag, with the dead code deleted, could not have woken it.

Shard Garden's flag rules

Every flag gets an owner, a kind and an expiry date. A release flag is removed within two weeks of reaching every player. The pipeline fails a build that contains a flag past its date.

Every flag has a new name. An old name is never reused.

Quick check

Reusing an old flag's name for a new feature is safe once the old feature is switched off.

Show the answer

False

Code that still reads the old flag may be running somewhere. Knight's eighth server was exactly that.

When it breaks anyway

Even with canaries, rollbacks and tidy flags, something will break. When it does, a team needs roles, not a crowd.

Bolt says: Next: incident command.

Fact-checked 12 October 2026