Gradient Descent

Gradient descent finds a function's minimum by repeatedly taking a small step in the direction that decreases it fastest — the negative gradient, from the Calculus Review course. For a single variable, "the gradient" is just the derivative:

def f_prime(x):
    return 2 * x   # derivative of f(x) = x^2

x = 10.0
learning_rate = 0.1
for _ in range(50):
    x -= learning_rate * f_prime(x)
print(x)   # converges toward 0 -- the true minimum of x^2

Every step shrinks x toward the minimum, because f_prime(x) shrinks toward 0 as x approaches it — the steps naturally get smaller as you get closer.

For a multivariable function, replace "the derivative" with "the gradient" (from Calculus Review) — same algorithm, applied to every coordinate simultaneously:

def grad_f(p):
    return [2*p[0], 2*p[1]]   # gradient of f(x,y) = x^2 + y^2

point = [5.0, 5.0]
lr = 0.1
for _ in range(50):
    g = grad_f(point)
    point = [point[i] - lr * g[i] for i in range(len(point))]
print(point)   # converges toward [0, 0]

The learning rate (lr) controls step size — too small and convergence takes forever; too large and it can overshoot the minimum and diverge instead of converging. Picking it well is most of the practical skill in using gradient descent.