Understanding Stochastic Gradient Descent
Full-batch gradient descent computes the average gradient over every training example before taking a single step. On a dataset of any size that is an expensive way to learn one number. Stochastic gradient descent replaces the average over all n examples with an average over a random sample of B of them, called a mini-batch, and takes a step immediately.
The key property is that this substitution introduces no bias. Because the mini-batch is drawn uniformly, the expectation of its gradient is exactly the full gradient. A small batch does not point somewhere systematically different; it points in the right direction with error added. This matters for the intuition about batch size: the question is never whether a small batch is "wrong", only how much noise the run can tolerate.
What a larger batch buys is precision, and it buys it slowly. Averaging independent estimates reduces their spread as the square root of the count, so quadrupling the batch halves the noise. Meanwhile the cost of a step grows linearly in the batch. That exchange rate - linear cost, square-root benefit - is the entire argument for small batches and many steps, and it is why mini-batch SGD rather than full-batch descent trains essentially every modern model.
The noise is not free. With a fixed step size the iterate never settles at the optimum: each step contracts toward it and also injects fresh sampling noise, and at some radius the two balance. The run reaches a stationary distribution - a ball around the minimum - and stays there. Shrinking the step shrinks the ball only as its square root, so halving the learning rate buys about a 30% reduction in the residual error. The fix is not a smaller constant step but a decaying one.
How to Calculate
w_{t+1} = w_t − η_t · (1/B) Σ_{i ∈ B_t} ∇ℓ(w_t; x_i, y_i)
where
- w_t
- the parameters at step t
- η_t
- the learning rate, possibly decaying with t
- B_t
- the mini-batch drawn at step t
- B
- the mini-batch size
- ∇ℓ
- the gradient of the per-example loss
Example of Stochastic Gradient Descent
On a least-squares problem with 400 points and two parameters, the full gradient at the origin w = (0, 0) is (−3.416495, 0.179597). Averaging 20,000 independent mini-batches at that same point gives a mean error of 0.032 at B = 8 and 0.002 at B = 128 - Monte-Carlo residue, converging to zero, which is what unbiasedness looks like when measured.
The spread behaves differently. The typical size of the noise is 1.4382 at B = 8 and 0.3027 at B = 128: sixteen times the batch for a factor of 4.75 in precision, close to the √16 = 4 that independent averaging predicts, and above it because the batches are drawn without replacement from only n = 400 points, which predicts 4√(392/272) = 4.80.
Running the same problem to 120,000 steps with a fixed step size shows the floor. The root-mean-square distance from the optimum settles at 0.109343 for η = 0.20 and 0.026497 for η = 0.0125. Fitting across that sixteenfold range gives an exponent of 0.5090 - the radius scales as √η. Replacing the constant step with η_t = 0.20/(1 + t/500) brings the same run to 0.009909, eleven times closer, with no change to the data or the initial rate.
Advantages and Disadvantages
Pros
- Makes progress after a handful of examples rather than a full pass over the data.
- Scales to datasets that do not fit in memory, since only a batch is ever loaded.
- The gradient noise helps the trajectory escape saddle points, which dominate high-dimensional loss surfaces.
Cons
- With a constant step it never converges, only settles into a noise ball.
- The residual error falls only as the square root of the step size, a poor exchange rate.
- Adds batch size to the list of hyperparameters that interact with the learning rate.
Frequently Asked Questions
Is "stochastic gradient descent" one example per step or a mini-batch?
Strictly, the original method uses one example per step. In practice the name is used for the mini-batch version, with batches of tens to thousands, which is what every deep learning framework implements. The mathematics is the same; only the amount of noise differs.
Why not just use a very small constant learning rate?
Because the residual error scales as the square root of the step size, so a sixteenfold reduction buys only a fourfold improvement - and it slows the approach by the same sixteenfold factor. A decaying schedule gets both: large steps early to travel, small steps late to settle.
Does a bigger batch always train better?
It trains with less noise per step, but the noise falls only as the square root while the cost per step grows linearly, so the same compute usually goes further as more, noisier steps. Very large batches also need the learning rate scaled up to compensate, and the gradient noise they remove is part of what helps generalisation.
The Bottom Line
Stochastic gradient descent works because a mini-batch gradient is the full gradient plus noise, not a different gradient. Everything characteristic about it - the cheapness of a step, the noise floor a constant step settles into, and the schedule that removes the floor - follows from that one fact.