Gradient Descent

Tap it to roll downhill. Hands-on: drop the ball anywhere; it rolls into a valley.

Gradient descent is how most neural networks learn. Picture the network's error as a hilly landscape: the lower the ground, the better its answers. At each step, it works out the gradient, which points toward the steepest increase, and steps in the opposite direction. The learning rate sets how big each step is, and momentum lets it keep some speed from step to step.

Tap it: the ball takes 21 hopping steps downhill, leaving a trail of dots, overshoots the valley and settles in it. In the Splash tab, set the Learning rate: too low, and it creeps down the slope; too high, and it bounces from wall to wall; just right, and it settles. This prepared path illustrates the three settings; descent is not guaranteed to find the lowest valley of every landscape. With ✋ Hands-on on, press on the landscape and the ball comes to your finger; let go and it rolls downhill by the slope, overshoots a little and settles in the nearest valley, which is not always the deepest one.

The paths are computed from a real loss function, finite-difference gradients, and momentum. They closely match independently derived analytic paths. They are then replayed, with hard boundaries and different momentum as well as learning rate.

1. Close

   The ball follows negative loss gradients with momentum.¹²

   All 22 points in each of the three paths agree with analytic-gradient trajectories to absolute error below 0.000006 in position and plotted height.

2. Correct

   Momentum carries part of the previous update.²

   The presets use learning rate and momentum pairs 0.035/0.3, 0.17/0.5, and 0.95/0.

3. Simplified

   The three paths are a limited example of learning-rate effects.¹²

   The large step is clipped at the walls, and each choice can also change momentum; the graphic is not a guarantee of convergence on arbitrary losses.

1. 1Gradient descent, Dive into Deep Learning authors
2. 2Momentum, Dive into Deep Learning authors

Simplified

The path is computed once and replayed, not optimized during each tap. Positions are clipped to a fixed square. The learning-rate choices also change momentum, so they are not a controlled comparison of step size alone.

Made in Splashery. Its code is MIT.