A/B Testing

An A/B test randomly splits users into two groups — control (the current experience) and variant (the change being tested) — and compares an outcome metric between them, usually a conversion rate: the fraction of visitors who did the thing you care about (bought, signed up, clicked).

def conversion_rate(conversions, visitors):
    return conversions / visitors

control = conversion_rate(50, 1000)     # 0.05 -- 5%
variant = conversion_rate(65, 1000)      # 0.065 -- 6.5%

A variant converting better in your sample doesn't automatically mean it's actually better — with enough randomness, one group can beat the other by chance alone, especially with a small sample. That's exactly the hypothesis-testing machinery from earlier in this course: treat "no real difference" as the null hypothesis, compute a p-value for the observed gap, and only trust the result once it's small enough (and the sample was large enough) to rule out chance as the likely explanation.

Two failure modes to watch for in practice: peeking (checking results early and stopping as soon as they look good, which inflates false positives) and too many simultaneous variants (testing 20 button colors means roughly 1 will look "significant" by pure chance even if none actually work).