Understanding Gradient Descent
Gradient descent answers a mechanical question: given a function that measures how wrong a model is, which way should the parameters move to make it less wrong? The gradient of that loss function, the vector of its partial derivatives with respect to each parameter, points in the direction in which the loss increases fastest. Stepping in the exact opposite direction therefore reduces the loss fastest, at least locally.
The algorithm is the loop that follows from this: compute the gradient at the current parameters, take a small step against it, and repeat. The step size is controlled by a single number, the learning rate. This one hyperparameter carries a lot of weight. Set it too small and the model takes an impractical number of steps to get anywhere; set it too large and the steps overshoot the minimum, so the loss oscillates or diverges outright.
The method is local. It follows the slope beneath the current point and has no view of the wider landscape, so it converges to a local minimum. For a convex loss surface, which has a single basin, that local minimum is the global one. For a neural network the surface is not convex, and the practical finding is that this matters far less than the theory once suggested: the minima reached from different starting points tend to generalize comparably well.
In practice the gradient is almost never computed over the entire training set at once, which would be prohibitively expensive. Stochastic gradient descent estimates it from a randomly drawn mini-batch. The estimate is noisier, but each step is far cheaper, and the noise itself is useful, helping the trajectory escape saddle points and narrow valleys. Modern optimizers such as momentum and Adam build on this same skeleton, adding a memory of past gradients to smooth the path.
How to Calculate
θ ← θ − η ∇L(θ)
where
- θ
- the parameters being optimized
- η
- the learning rate, the step size
- ∇L(θ)
- the gradient of the loss L with respect to θ
- ←
- assignment: the new value replaces the old each iteration
Example of Gradient Descent
Take the simplest possible loss, f(w) = (w − 3)², whose minimum is obviously at w = 3. Its derivative is f′(w) = 2(w − 3), so the update rule is w ← w − η · 2(w − 3). Start deliberately far away at w = 10 with a learning rate of η = 0.1.
The first five steps give w = 8.6000, 7.4800, 6.5840, 5.8672, 5.2938, with the loss falling 31.36 → 20.07 → 12.85 → 8.22 → 5.26. Each step closes 20% of the remaining distance to 3, so progress is fast at first and then slows as the gradient shrinks near the minimum.
The learning rate is not a free choice. For this function, any η between 0 and 1 converges; at exactly η = 1 the iterate jumps to the mirror-image point and oscillates forever without improving; above 1 it diverges. Every loss surface has an equivalent stability threshold set by its curvature, which is why a learning rate that works for one model can destroy training on another.
Advantages and Disadvantages
Pros
- Requires only first derivatives, so it scales to models with billions of parameters.
- Memory-cheap: no need to build or invert a matrix of second derivatives.
- The mini-batch form works on datasets far too large to fit in memory.
Cons
- Converges only to a local minimum, with no guarantee it is the global one for non-convex losses.
- Highly sensitive to the learning rate, which usually has to be tuned empirically.
- Struggles in narrow, elongated valleys, where it zig-zags rather than travelling along the floor.
Frequently Asked Questions
Why subtract the gradient rather than add it?
The gradient points toward steepest increase of the loss. Since the goal is to make the loss smaller, the step goes in the opposite direction. Adding the gradient would perform gradient ascent, which is exactly what you want when maximizing an objective instead of minimizing an error.
What is the difference between batch, stochastic, and mini-batch gradient descent?
Batch gradient descent computes the gradient over the entire training set for every step: accurate but slow. Stochastic gradient descent uses a single example per step: fast but very noisy. Mini-batch, typically tens to hundreds of examples, sits between them and is what essentially all deep learning uses in practice.
Does gradient descent get stuck in local minima when training neural networks?
Less than the intuition from low-dimensional pictures suggests. In very high-dimensional parameter spaces, saddle points are far more common than poor local minima, and the noise inherent in mini-batch gradients helps the trajectory move away from them.
The Bottom Line
Gradient descent is the engine underneath nearly all modern model fitting: measure the slope of the error, step downhill, repeat. Its simplicity is what allows it to scale to enormous models, and its two persistent difficulties, choosing a learning rate and coping with awkward loss geometry, are what most of the optimizer literature exists to address.