Machine Learning & AI
Intermediate

SGD with Momentum

Accelerates gradient descent by accumulating a velocity of past gradients.

Formula

v:=βv+(1−β)∇J;    θ:=θ−αvv := \beta v + (1-\beta)\nabla J;\;\; \theta := \theta - \alpha v

Variables

vVelocity
\betaMomentum (~0.9)
\alphaLearning rate

Example

beta=0.9 keeps 90% of prior velocity

Did You Know?

Momentum helps the optimizer roll through small local dips and ravines like a ball gaining speed.

optimizationmomentumtraining

Share this formula

More in Machine Learning & AI

View all

Linear Regression Model

Basic

Predicts a continuous value as a weighted sum of input features plus a bias.

Gradient Descent Update

Basic

Iteratively moves parameters in the direction that most reduces the loss.

ReLU Activation

Basic

Rectified Linear Unit: outputs the input if positive, else zero.

Leaky ReLU

Basic

A ReLU variant that lets a small gradient flow for negative inputs.

Tanh Activation

Basic

Squashes input to the range (-1, 1); zero-centred activation.

Softmax

Basic

Turns a vector of scores into a probability distribution that sums to 1.