Model the goal before solving it

A solver only sees the score and the allowed choices, so a precise objective is a mathematical version of what success means.

An unconstrained minimization problem is written minₓ f(x). The vector x contains decisions and f maps each decision to one comparable cost.

A useful objective may combine data fit and regularisation: empirical loss measures mistakes, while a penalty discourages overly complex or risky choices.

Scaling matters. Adding quantities with wildly different units or magnitudes can make one term dominate for accidental reasons.

Decision variables must include only quantities the solver is allowed to change; measured data and fixed requirements belong in the objective as inputs, not choices.

A proxy loss should be tested for unintended shortcuts because the optimizer cannot know the human meaning that was omitted from its formula.

Regularisation strength creates a trade-off between fit and preference, so both terms should be scaled and reported rather than silently mixed across incompatible units.

Follow the negative gradient

The gradient points uphill, so subtracting a small multiple gives the locally fastest first-order decrease.

Gradient descent updates xₖ₊₁=xₖ−η∇f(xₖ), where η>0 is the learning rate.

Too small a learning rate wastes steps; too large a rate can bounce across a valley, increase the objective, or diverge.

Stopping rules can use a small gradient norm, small objective improvement, a step budget, or validation performance. Each detects a different kind of 'done'.

The update comes from the local model f(x+Δ)≈f(x)+∇f^TΔ; among equal-length Euclidean steps, choosing Δ opposite the gradient lowers that linear term most.

On a quadratic, the largest curvature limits a stable constant learning rate because a step safe along a flat direction may overshoot along a steep one.

Objective decrease, gradient norm, parameter movement, and validation performance are separate stopping signals and can disagree near plateaus or noisy batches.

Practical deep dive for this lesson

How a controller learns a useful setting instead of guessing

Optimization begins by turning success into a measurable objective. Gradient descent then uses local slope to improve one setting at a time, while repeated checks make sure the score really means what we want.

Suppose a greenhouse controller chooses heater power x. Too little misses the target temperature, while too much wastes energy and may overshoot.

An objective f(x) combines those consequences into one number for each allowed setting. Smaller values must consistently mean a better decision.

Choosing the variables and score is not clerical work: if energy cost is omitted, the mathematical optimum may simply use maximum heat.

The derivative f′(x) estimates how the score changes for a small positive adjustment. A positive derivative means increasing x raises cost locally, so we should move left.

Gradient descent applies x_new=x−ηf′(x). Subtracting the derivative chooses the downhill direction, while η decides how far to trust the local slope.

A tiny η gives cautious but slow improvement. An oversized η can jump across the valley and land at a worse score or oscillate forever.

Recording f before and after each step makes the process inspectable. If the objective rises repeatedly, reduce the step or reconsider scaling and derivative correctness.

A small derivative is a stopping clue, not proof of success. The point may be a maximum, saddle, flat plateau, or boundary-limited solution.

The controller must also test the chosen setting on new days because weather variation can make an objective fitted to past measurements overconfident.

This same loop—define, differentiate, step, measure, validate—trains regression models, tunes physical designs, and calibrates parameters in simulations.