Training with Gradients and Backpropagation

Computer ScienceMachine LearningNeural NetworksOptimization Guide August 8, 2026

Training a neural network means choosing parameters that reduce a loss function. Gradient descent chooses the update; backpropagation computes the derivatives needed for that update.

Start with one parameter

Consider the model y^=wx\hat y=wx and squared loss

L(w)=(wx−y)2.L(w)=(wx-y)^2.

The derivative is

dLdw=2(wx−y)x.\frac{dL}{dw}=2(wx-y)x.

If x=3x=3, y=12y=12, and w=2w=2, then y^=6\hat y=6, the error is −6-6, and

dLdw=2(−6)(3)=−36.\frac{dL}{dw}=2(-6)(3)=-36.

Gradient descent updates

wnew=w−ηdLdw,w_{\text{new}}=w-\eta\frac{dL}{dw},

where η\eta is the learning rate. With η=0.01\eta=0.01, the new weight is 2.362.36. Because the derivative was negative, increasing ww locally reduces the loss.

The chain rule through a network

Suppose a computation has intermediate values

z=wx+b,h=σ(z),y^=vh,z=wx+b,\qquad h=\sigma(z),\qquad \hat y=vh,

followed by loss L(y^,y)L(\hat y,y). The derivative with respect to ww is a product of local derivatives:

∂L∂w=∂L∂y^∂y^∂h∂h∂z∂z∂w.\frac{\partial L}{\partial w} =\frac{\partial L}{\partial \hat y} \frac{\partial \hat y}{\partial h} \frac{\partial h}{\partial z} \frac{\partial z}{\partial w}.

Backpropagation applies this chain rule efficiently, reusing derivatives for branches that share earlier computations. It is not a separate learning rule and it does not decide the direction of the update. It produces gradients; an optimizer uses them.

Learning rate and stability

A very small learning rate can make progress slow. A very large one can jump across a minimum or cause the loss to grow. A useful training graph shows loss against update step. Smooth decline is not required, especially with mini-batches, but repeated explosion or no movement signals a problem.

Feature scale also matters. If one input varies near 10−310^{-3} and another near 10410^4, their gradients can have very different sizes. Standardization often makes the optimization landscape easier to navigate.

Training loss is not the final score

Measure performance on data not used for parameter updates. A validation set helps select settings such as learning rate and model size. A test set estimates final performance after those choices are fixed.

Watch for a widening gap: training loss continues to improve while validation performance worsens. This is evidence of overfitting. Possible responses include collecting more representative data, reducing model capacity, adding regularization, or stopping earlier.

A computation-graph activity

Draw a graph for L=(wx−y)2L=(wx-y)^2. Students compute the forward values from left to right, then attach local derivatives and multiply them from right to left. Repeat with a ReLU node. If its input is negative, its derivative is zero, so the gradient through that path is zero.

The activity separates three ideas that are often blurred together:

  • the forward pass computes a prediction and loss;
  • backpropagation computes derivatives;
  • the optimizer changes parameters.

Keeping those roles distinct makes more advanced training methods much easier to understand.

← Back to All Resources