From regression to logits and cross-entropy

Linear regression predicts a number, logistic regression predicts a binary probability, and softmax extends the same score-to-probability idea to many classes.

Linear regression uses ŷ=wᵀx+b and commonly fits squared error. Logistic regression feeds the score through σ(z)=1/(1+e⁻ᶻ), then uses binary cross-entropy; both are simple models that reveal the full prediction-loss-gradient loop.

For logits z, softmax assigns pᵢ=exp(zᵢ) ÷ Σⱼexp(zⱼ); subtract max(z) before exponentiating for numerical stability.

For a one-hot target y, loss L=−Σᵢyᵢ log pᵢ=−log p(correct class).

Cross-entropy is justified as negative log-likelihood for a categorical observation; label noise and distribution shift still matter.

A model first converts input features into raw scores. Logistic sigmoid handles two classes; softmax exponentiates and normalizes many logits so their probabilities are nonnegative and total one.

Cross-entropy reads the probability assigned to the correct answer. A wrong but uncertain forecast receives moderate loss, while a wrong forecast made with near-certainty receives a large loss, which pushes training to correct confident errors.

Subtracting the largest logit before exponentiation leaves all softmax ratios unchanged because the same constant cancels. It prevents enormous exponentials and is therefore a mathematical equivalence with better numerical behavior.

Backpropagation and gradient updates

The chain rule passes responsibility backward through every operation used in the forward prediction.

A computational graph stores intermediate values, then reverse-mode automatic differentiation accumulates vector–Jacobian products.

Gradient descent updates θ←θ−η∇L; η is the learning rate, and stochastic mini-batches give noisy gradient estimates.

Gradients can vanish or explode through many multiplications; initialization, residual paths, normalization, and clipping help in different ways.

Backpropagation applies the chain rule from the loss toward earlier operations. Each parameter receives a derivative measuring how an extremely small increase would change the current batch loss.

Gradient descent moves in the negative-gradient direction because the gradient points toward local increase. The learning rate decides how far to trust that local linear picture before recomputing at a new point.

Mini-batches replace the full-data gradient with a noisy estimate that is much cheaper to compute. Shuffling, optimizer state, clipping, and schedules control this noise and scale, but held-out evaluation is still needed to detect overfitting.

Practical deep dive for this lesson

Follow one gradient through a tiny classifier

A concrete forward and backward pass shows how prediction error becomes a parameter update instead of remaining an abstract arrow.

For binary logistic regression, compute z=wx+b, p=sigmoid(z), and loss from the true label y. The derivative with respect to z simplifies to p−y.

The chain rule gives dL/dw=(p−y)x and dL/db=p−y. The feature value scales how strongly this example moves its weight, while the sign says which direction increases the correct-class probability.

A batch averages such contributions before the optimizer updates. Feature scaling, regularization, and learning rate alter training dynamics, so gradient arithmetic should be checked on a tiny hand-calculated case.