Hypothesis Testing

Hypothesis testing is how you decide whether an observed difference is real or just noise. You start with a null hypothesis (H₀ — "there's no real effect, any difference is chance"), then ask: how surprising is the data we actually observed, if H₀ were true?

A z-test compares a sample mean to a claimed population mean, scaled by how much sample means naturally vary:

import math

def z_score(sample_mean, pop_mean, pop_std, n):
    standard_error = pop_std / math.sqrt(n)
    return (sample_mean - pop_mean) / standard_error

A z-score of 2.0 means the sample mean is 2 standard errors away from what you'd expect under H₀ — the larger |z|, the more surprising the result.

The p-value converts a z-score into a probability: "if H₀ were true, what's the chance of seeing a result at least this extreme?" A small p-value (conventionally below 0.05) is usually treated as evidence against H₀.

import statistics

def two_sided_p_value(z):
    return 2 * (1 - statistics.NormalDist().cdf(abs(z)))

NormalDist().cdf(z) gives P(Z ≤ z) for the standard normal — the "two sided" part accounts for the effect being unusually large in either direction, not just the one you happened to observe.

A p-value is NOT "the probability the null hypothesis is true" — that's one of the most common misinterpretations. It only ever describes how surprising the data is, assuming H₀ is true.