Chain Rule for Backprop

The chain rule tells you how to differentiate a function built by composing two others: d/dx[f(g(x))] = f'(g(x)) · g'(x) — the outer function's derivative, evaluated at the inner function's OUTPUT, times the inner function's derivative.

# g(x) = 3x + 1,  g'(x) = 3
# f(u) = u**2,     f'(u) = 2u
# f(g(x)) = (3x + 1)**2

def d_dx(x):
    g = 3*x + 1
    g_prime = 3
    f_prime_at_g = 2 * g   # f'(u) = 2u, evaluated at u = g(x)
    return f_prime_at_g * g_prime

This is EXACTLY what backpropagation does, just at a much larger scale. A neural network is a long chain of compositions — layer 1 feeds into layer 2, which feeds into layer 3, and so on, ending in a loss function. Training the network means computing how the loss changes with respect to EVERY weight in EVERY layer — and the chain rule is the only tool that makes that tractable: you compute the derivative at the output, then multiply backward through each layer's local derivative, one composition at a time.

# y = (w*x + b)**2 -- a tiny "one linear layer + squared loss" network
# dy/dw: outer function is (...)**2, inner is w*x + b
def dy_dw(w, x, b):
    inner = w*x + b
    d_inner_d_w = x            # d/dw[w*x + b] = x
    d_outer_d_inner = 2*inner  # d/du[u**2] = 2u
    return d_outer_d_inner * d_inner_d_w

"Backpropagation" is really just this same pattern applied layer after layer, computed efficiently by reusing intermediate results instead of recomputing them — the calculus doesn't change at any scale, only the bookkeeping does.