Derivatives
18 min
The derivative of a function at a point measures how fast it's changing right there — the slope of the line tangent to the curve at that exact point. Every gradient-based ML technique (gradient descent, backpropagation) is built entirely out of derivatives.
For a function like f(x) = x², calculus gives you an exact formula:
f'(x) = 2x. But you don't always have (or need) the exact formula —
you can estimate a derivative numerically, directly from the function
itself.
This is the 'central difference' formula: nudge x slightly in both directions, see how much f changes, divide by the total nudge. As h shrinks toward 0, this converges to the true derivative.
The central difference formula (f(x+h) - f(x-h)) / (2h) works for ANY
function you can call — you never need its symbolic derivative. That's
exactly the technique behind gradient checking, a standard way to
verify a hand-derived (or autograd-computed) gradient is correct: compute
it both ways and confirm they match.
Picking h matters: too large and the estimate is inaccurate (it's
measuring the average slope over too wide an interval, not the
instantaneous one); too small (smaller than about 1e-8 for standard
floating-point numbers) and rounding error dominates. 1e-5 is a
reasonable default for most functions.
Write `numerical_derivative(f, x, h=1e-5)`, estimating f's derivative at `x` using the central difference formula `(f(x+h) - f(x-h)) / (2*h)`. Round the result to 4 decimal places.
Write `is_increasing(f, x, h=1e-5)`, returning True if f's derivative at `x` is positive (the function is increasing there), False otherwise.
What does a function's derivative at a point represent?
You can estimate a function's derivative numerically using the central difference formula, without needing its symbolic derivative.