RMSProp
Scales the learning rate by a running average of recent squared gradients.
Formula
Variables
Example
Damps steps in steep directions
How it works
RMSProp keeps an exponential moving average of the squared gradients, so directions with consistently large gradients get a smaller effective step size. This helps because in many optimization problems, some parameters have much steeper curvature than others; dividing by a running root-mean-square of past gradients prevents the updates from oscillating wildly in those directions. The decay factor \(\gamma\) controls how much recent gradients influence the average, so the method adapts to the local scale of the loss surface.
When to use it
Use RMSProp when training models with noisy or poorly scaled gradients, especially in nonconvex problems like neural networks. It is especially helpful when plain gradient descent feels unstable or requires very careful learning-rate tuning.
Worked example, step by step
Suppose the previous running average of squared gradients is \(E[g^2]_{t-1} = 0.40\), the current gradient is \(g_t = -2\), and \(\gamma = 0.9\). Step 1: Square the current gradient: \(g_t^2 = (-2)^2 = 4\). Step 2: Weight the old average: \(\gamma E[g^2]_{t-1} = 0.9 \times 0.40 = 0.36\). Step 3: Weight the new squared gradient: \((1-\gamma)g_t^2 = 0.1 \times 4 = 0.40\). Step 4: Add them: \(E[g^2]_t = 0.36 + 0.40 = 0.76\). Final answer: the updated running average of squared gradients is \(0.76\), so the optimizer will use a smaller effective learning rate than it would in a flatter direction.
Common mistakes to avoid
- Using the raw gradient instead of the squared gradient in the moving average
- Forgetting that \(\gamma\) close to 1 gives more memory of past gradients
- Treating \(E[g^2]_t\) as the parameter update itself rather than as a scaling term for the step size
Did You Know?
RMSProp was famously introduced by Geoffrey Hinton in a Coursera lecture, not a paper.
Share this formula
More in Machine Learning & AI
View allLinear Regression Model
BasicPredicts a continuous value as a weighted sum of input features plus a bias.
Gradient Descent Update
BasicIteratively moves parameters in the direction that most reduces the loss.
ReLU Activation
BasicRectified Linear Unit: outputs the input if positive, else zero.
Leaky ReLU
BasicA ReLU variant that lets a small gradient flow for negative inputs.
Tanh Activation
BasicSquashes input to the range (-1, 1); zero-centred activation.
Softmax
BasicTurns a vector of scores into a probability distribution that sums to 1.