A/B tests: the statistics

Play this lesson

B wins, or does it?

After a week of the Invite test, 40.5% of group B joined a co-op garden, against 39.5% in group A. Kai says: "B wins. Ship it!"

Theo isn't sure: two random groups never behave exactly alike. Start from the null hypothesis: the button makes no difference.

Firebase's docs describe the p-value as the probability of seeing data as extreme as the results if there is no true difference. A small p-value means luck alone would rarely produce results like these.

Firebase A/B Testing uses a significance level of 0.05: a p-value below 0.05 counts as a statistically significant difference.

What 0.05 means

The NIST handbook, from the US standards agency, explains: a significance level of 0.05 means the null hypothesis is rejected 5% of the time when it is in fact true.

So if the studio tests 20 changes that really do nothing, about one will still look like a winner. That is a false positive. The other mistake, missing a real difference, is a false negative.

Quick check

A p-value of 0.03 means there is a 3% chance that the new button makes no difference.

Show the answer

False

The American Statistical Association says p-values do not measure the probability that a hypothesis is true. 0.03 means data like this would be rare if the button made no difference.

What a p-value can't tell you

In 2016 the American Statistical Association published six principles about p-values. Three of them:

A tiny p-value can come with a difference too small to matter.

The confidence interval

Firebase also shows a 95% confidence interval for the difference between the groups: a range of plausible values, not a single number. If the range includes 0, no significant difference was detected.

An interval shows the size of the effect and the uncertainty together: "+1.0 points, somewhere between −0.4 and +2.4" is much more honest than "B wins".

Read the result

invite test, joined co-op in 7 days
Group  Players   RateA        9,800  39.5%B        9,750  40.5%B minus A: +1.0 points95% CI: -0.4 to +2.4

What can Theo say?

  1. No significant difference yet: the interval includes 0
  2. B wins by 1.0 point
  3. A wins, because the interval goes below 0
Show the answer

No significant difference yet: the interval includes 0

The true difference could be anything from a small loss to a gain of 2.4 points, including no difference at all. More players would narrow the interval.

How many players

The NIST handbook's sample size formula shows that the number of players needed grows with the square of (spread ÷ smallest difference worth detecting). Halve the difference you want to detect, and you need four times the players.

Firebase recommends at least two weeks for a typical Remote Config experiment. Theo also works out the number of players before the test starts.

Do the math

Theo needs 10,000 players per group to detect a 2-point change. About how many are needed to detect a 1-point change?

  1. 40,000 per group
  2. 20,000 per group
  3. 5,000 per group
Show the answer

40,000 per group

Halving the difference doubles (spread ÷ difference), and the sample size grows with its square: 2 × 2 = 4 times as many players.

Don't peek and stop

Kai checks the test every morning and wants to stop the day it looks significant. In 2010, Evan Miller, who writes about A/B testing statistics, showed why not: peek ten times, stopping when it looks significant, and what you think is 1% significance is actually 5%.

His fix: decide the sample size in advance and wait until it is reached. Methods built for checking early, called sequential tests, exist too.

Bolt says: Next: lifetime value, and what makes a player worth keeping.

Fact-checked 11 October 2026