Constrained Optimization

So far, every optimization problem has been unconstrained — minimize f(x) over ALL possible x. Real problems usually have constraints: minimize risk subject to a minimum expected return, minimize cost subject to a budget, and so on.

The simplest constrained problems can be solved by substitution: if x + y = 10 is required, substitute y = 10 - x and you're back to an ordinary unconstrained problem in one variable:

# minimize x^2 + y^2 subject to x + y = 10
# substitute y = 10 - x:
# minimize x^2 + (10 - x)^2   -- now just an unconstrained 1D problem

By symmetry, the minimum of x² + (10-x)² occurs at x = y = 5 — split evenly, since squaring penalizes extremes more than the constraint rewards concentrating in one variable.

Substitution doesn't work for INEQUALITY constraints like x >= 5 — you can't "solve for" an inequality the way you can an equation. The penalty method handles this instead: add a term to the objective that's zero when the constraint is satisfied, and grows large (quadratically, usually) when it's violated:

def objective(x, penalty_weight=100):
    violation = max(0, 5 - x)          # 0 if x >= 5, positive otherwise
    return x**2 + penalty_weight * violation**2

Minimizing this modified objective with ordinary gradient descent naturally avoids the infeasible region — any attempt to go below x = 5 gets punished so heavily that gradient descent steers back toward x = 5, converging to the constrained optimum without ever solving the constraint algebraically.