Hypotheses & Baselines
18 min
Before building anything sophisticated, every ML project needs a baseline — the simplest reasonable approach (a constant prediction, a simple heuristic, or a basic model) that any "real" model must beat to justify its added complexity. A new model's metric being HIGHER than the baseline isn't automatically meaningful, though — training and evaluation both have some inherent noise, so a proper comparison checks for an improvement large enough to trust, not just any improvement at all.
def significant_improvements(results, baseline_metric, min_improvement=0.02):
winners = [(name, m) for name, m in results.items() if m - baseline_metric >= min_improvement]
winners.sort(key=lambda pair: -pair[1])
return [name for name, _ in winners]
This is deliberately a simple heuristic (a fixed margin), not a rigorous statistical test -- a real experiment would follow this up with Statistics' hypothesis-testing/confidence-intervals modules to get an actual p-value or confidence interval on the difference, rather than eyeballing a fixed threshold.
Write `significant_improvements(results, baseline_metric, min_improvement=0.02)`: `results` maps model name to its metric. Return the names of models whose metric beats `baseline_metric` by at least `min_improvement`, sorted best-metric-first.
Why does `significant_improvements` require a MINIMUM improvement margin instead of just checking `model_metric > baseline_metric`?
You can filter and rank candidate models by whether they clear a meaningful improvement margin over a baseline, not just any positive difference.