Skip to Content

A/B Testing and Statistical Significance

When to use

Compare prompt A vs prompt B or model v1 vs v2 — know if the win is luck.

Analogy

If you roll 20 dice, the chance of one showing a 6 is high. If you A/B test 20 prompt variants, one will look like a winner purely by chance — that is the multiple-comparisons problem.

Data-flow diagram

   H0: no difference between A and B
   significance: p < alpha (typical alpha 0.05)
   Bonferroni:  alpha / m   (m = number of comparisons)

Deep explanation

An A/B test runs two strategies on randomly-assigned users and compares a metric. The p-value is the probability of seeing the observed difference (or larger) if there were actually no difference. For ML model swaps always run for at least one full business cycle to capture time-of-week effects. Use Bonferroni or Benjamini-Hochberg for multiple comparisons.

Examples

Example 1

from scipy.stats import ttest_ind
import numpy as np
a = np.array([0.10, 0.12, 0.09, 0.11, 0.13, 0.10, 0.12])
b = np.array([0.14, 0.15, 0.13, 0.16, 0.14, 0.15, 0.17])
stat, p = ttest_ind(a, b)
print(f't={stat:.3f}  p={p:.3g}')

scipy.stats.ttest_ind gives two-sample t — assumes normal-ish data.

Example 2

from scipy.stats import norm
def required_n(effect_size, alpha=0.05, power=0.8):
    z_alpha = norm.ppf(1 - alpha/2)
    z_beta  = norm.ppf(power)
    return int(2 * ((z_alpha + z_beta) / effect_size) ** 2)
print('required sample per arm for 0.05 effect:', required_n(0.05))

Power analysis tells you sample size for a given effect size — run BEFORE the test.

Example 3

from statsmodels.stats.multitest import multipletests
pvals = [0.001, 0.04, 0.06, 0.20, 0.50]
reject, p_adj, _, _ = multipletests(pvals, alpha=0.05, method='fdr_bh')
print('BH-rejected:', reject, 'adjusted-p:', p_adj)

Benjamini-Hochberg controls FDR — less conservative than Bonferroni.

Common mistake

Stopping the A/B test the first time p dips under 0.05. Peeking inflates the false-positive rate. 20 prompt variants without Bonferroni — the winner is almost certainly luck.

Key takeaway

Run for a full business cycle. Pre-register sample size with power analysis. Use Bonferroni or Benjamini-Hochberg for multiple comparisons.

Production Failure Playbook

Failure scenario 1: peeking-at-p-value

Failure scenario 2: many-variants-no-correction