A/B tests: the design

Play this lesson

A bigger Invite button

Remy designs a big Invite button for the victory screen after a player's first garden. Kai wants to ship it to everyone. Theo, the studio's data analyst, wants to test it first.

An A/B test splits players at random: group A keeps the old screen, group B gets the new one, and the studio compares them.

Microsoft's experimentation team says the hypothesis should state the expected effect, and the metrics must be able to prove or disprove it. Theo writes it first: "The big Invite button will raise the share of new players who join a co-op garden within 7 days." Written first, it can't quietly change after the results arrive.

One goal, some guardrails

Firebase A/B Testing asks for one goal metric, which decides the winner, and can track up to five more.

Microsoft's team describes guardrail metrics: things you don't want to get worse, but don't expect to improve, like crash rate or load time.

Theo's guardrails: crash-free sessions, join time at p95, and players blocking or reporting invites. A button that wins on co-op joins but floods players with unwanted invites doesn't ship.

Pick the goal

Which is the best goal metric for the Invite button test?

  1. Share of new players who join a co-op garden within 7 days
  2. Number of taps on the Invite button
  3. Crash rate on the victory screen
Show the answer

Share of new players who join a co-op garden within 7 days

Taps can rise without anyone joining a garden, and the crash rate is a guardrail. The goal metric matches the hypothesis: players actually playing together.

What gets split

Microsoft's team says to choose the randomisation unit with care, thinking about whether the ID stays the same and about network effects, where one group affects the other.

Theo splits by player, so each player always sees the same version. Shard Garden is co-op, though: a player in group B can invite a friend in group A. For a change inside a garden, like a new shared shield, the unit would be the whole garden.

Quick check

Splitting by session is fine for the Invite test, as long as the split is 50/50.

Show the answer

False

By session, one player would see both buttons on different days, and their behaviour would mix both versions. Splitting by player keeps each player in one group.

Count only who could see it

Only new players who win a first garden ever see the victory screen. Counting everyone else waters the effect down.

Firebase's activation event handles this: only players who first triggered a chosen event, here first_garden_defended, are counted in the experiment.

On the web, Firebase notes, one person may land in different variants on different browsers. Theo keeps that in mind.

Check the split

A sample ratio mismatch (SRM) is when the groups' sizes differ from the planned split by more than chance explains. Microsoft found about 6% of its A/B tests had one.

Causes include bugs in how players are assigned, or a variant that crashes before it can log anything. Microsoft's advice: don't trust the results until the cause is found.

Read the numbers

invite test, day 5
Planned split   50% / 50%Players in A    10,212Players in B     8,940

What should Theo do?

  1. Find out why B has fewer players before trusting any result
  2. Read the goal metric as usual, the gap is small
  3. Move 636 players from A to B to even it out
Show the answer

Find out why B has fewer players before trusting any result

With almost 20,000 players, a gap this big is far beyond chance: a sample ratio mismatch. Maybe B crashes before logging. Until the cause is found, the results can't be trusted.

How many players, how long?

The test is designed. The next questions are about numbers: how big a difference is real, and when the test may stop.

Bolt says: Next: the statistics of A/B tests.

Fact-checked 11 October 2026