Gradients
18 min
For a function of several variables, each variable has its own partial derivative — how fast the function changes if you nudge JUST that one variable, holding the others fixed. The gradient collects all of them into a vector.
def f(p):
return p[0]**2 + p[1]**2 # p = [x, y]
# partial derivative w.r.t. x: nudge only p[0]
# partial derivative w.r.t. y: nudge only p[1]
This is exactly the extension of last module's numerical_derivative to
more than one input — nudge one coordinate at a time, holding the rest
fixed, and collect the results.
Each entry of the gradient only nudges ONE coordinate at a time -- that's the whole difference from the single-variable derivative from the previous module.
The gradient's DIRECTION points toward steepest increase — which is exactly why gradient DESCENT (covered in the next course, Optimization) moves in the OPPOSITE direction to minimize a function. Its MAGNITUDE (the same Euclidean length/L2 norm from the Linear Algebra course) tells you how steep that increase is — a large gradient magnitude means the function is changing rapidly at that point; a magnitude near zero means you're near a flat spot (possibly a minimum, maximum, or saddle point).
import math
magnitude = math.sqrt(sum(g**2 for g in grad))
That's the exact same magnitude computation from
linear-algebra/vectors-matrices — the gradient IS just a vector, so
every vector operation you already know applies to it directly.
Write `gradient(f, point, h=1e-5)`: `f` takes a list of numbers and returns a number; `point` is a list. Return the gradient — a list where each entry is the central-difference partial derivative with respect to that coordinate, rounded to 4 decimals.
Given the `gradient` helper below (already implemented), write `gradient_magnitude(f, point, h=1e-5)`, returning the gradient's Euclidean length (its L2 norm), rounded to 4 decimals.
What does the gradient of a multivariable function represent?
You can compute a multivariable function's gradient numerically, and know that its direction points toward steepest increase while its magnitude measures how steep.