SGD with Momentum
Accelerates gradient descent by accumulating a velocity of past gradients.
Formula
Variables
Example
beta=0.9 keeps 90% of prior velocity
How it works
Stochastic gradient descent with momentum adds a running average of past gradients, so parameter updates are guided not just by the current batch but by the recent direction of travel. The velocity term smooths out noisy step-to-step changes and helps the optimizer keep moving through shallow regions or small local bumps instead of zigzagging. In effect, gradients that consistently point the same way build up speed, while conflicting gradients are damped.
When to use it
Use momentum when training models with noisy or high-dimensional loss surfaces, such as neural networks, where plain SGD can be slow or unstable. It is especially helpful when gradients oscillate across narrow valleys or when you want faster convergence than standard SGD without moving all the way to adaptive methods.
Worked example, step by step
Given: β = 0.8, α = 0.05, initial v = 0, current gradient ∇J = 4 Step 1: Update velocity v := βv + (1 - β)∇J = 0.8(0) + 0.2(4) = 0.8 Step 2: Update parameter θ := θ - αv = θ - 0.05(0.8) = θ - 0.04 If the current parameter was θ = 2.50, the new value is θ = 2.50 - 0.04 = 2.46 Final answer: v = 0.8 and θ decreases by 0.04 to 2.46
Common mistakes to avoid
- Treating v as the gradient itself instead of a smoothed average of gradients
- Using a momentum value near 1 without adjusting the learning rate, which can make updates too sluggish or unstable
- Forgetting that the parameter update uses αv, not α∇J directly
Did You Know?
Momentum helps the optimizer roll through small local dips and ravines like a ball gaining speed.
Share this formula
More in Machine Learning & AI
View allLinear Regression Model
BasicPredicts a continuous value as a weighted sum of input features plus a bias.
Gradient Descent Update
BasicIteratively moves parameters in the direction that most reduces the loss.
ReLU Activation
BasicRectified Linear Unit: outputs the input if positive, else zero.
Leaky ReLU
BasicA ReLU variant that lets a small gradient flow for negative inputs.
Tanh Activation
BasicSquashes input to the range (-1, 1); zero-centred activation.
Softmax
BasicTurns a vector of scores into a probability distribution that sums to 1.