Gradient descent · Explore together

Choose the next step

Move a weight along a loss curve and investigate what the learning rate changes.

One person controls the experiment. The other predicts and questions. Swap roles after a trial.

wnew=w−ηdLdww_{\mathrm{new}}=w-\eta\frac{dL}{dw}

Squared loss L(w) = (2w − 6)² · minimum at w = 3

Scroll across the graph to compare the updates.

Weight wLoss L
Current weight 1Loss 16Gradient −16
    Try 0.10, then 0.25, then 0.35. Reset between trials.

    Make a prediction before taking the first step.

    Compare two experiments

    Which step sizes move closer to w = 3? Can a step cross the minimum and still improve the loss?

    Why can the updates overshoot?

    dL/dw = 8(w − 3), so wₙ₊₁ − 3 = (1 − 8η)(wₙ − 3). In this model, 0 < η < 0.25 converges; η = 0.25 repeats an equal-loss oscillation; larger rates move farther away.

    That conclusion belongs to this quadratic. A different loss landscape needs its own analysis.

    Compare your evidence

    What can you explain now?

    Ask your partner to restate your reasoning. An experiment suggests a pattern; your explanation should say why it holds and when it applies.

    1. Why does a negative gradient move the weight to the right?
    2. What changes when you double the learning rate?
    3. Does this one-variable quadratic tell us what every neural network will do?