Bias-Variance Tradeoff

A model's total expected error decomposes into three pieces: bias (error from a model too simple to capture the real pattern — underfitting), variance (error from a model so sensitive to the specific training data that it captures noise, not signal — overfitting), and irreducible error (noise in the data itself that no model could ever predict away).

def total_error(bias_squared, variance, irreducible_error):
    return bias_squared + variance + irreducible_error

Bias and variance trade off against each other as model complexity changes: a very simple model (like fitting a straight line to obviously curved data) has high bias, low variance. A very complex model (fitting every wiggle in the training data) has low bias, high variance. The best model complexity sits somewhere in between.

The gap between train and test error, and specifically the point where test error starts getting WORSE while train error keeps improving, is exactly how you detect overfitting has begun:

def detect_overfitting(train_errors, test_errors):
    for i in range(1, len(train_errors)):
        if test_errors[i] > test_errors[i - 1] and train_errors[i] < train_errors[i - 1]:
            return i   # complexity level where overfitting starts
    return -1

In practice, you'd pick the model complexity right BEFORE this point — complex enough to capture the real pattern (low bias), but not so complex it starts fitting noise (rising variance). This is exactly what cross-validation from the previous module is used FOR in practice: trying several complexity levels and picking whichever one cross-validates best, not whichever fits the training data best.