Machine Learning & AI

Intermediate

RMSProp

Scales the learning rate by a running average of recent squared gradients.

Formula

E[g^2]_t = \gamma E[g^2]_{t-1} + (1-\gamma)g_t^2

Variables

g_tCurrent gradient

\gammaDecay (~0.9)

Example

Damps steps in steep directions

Did You Know?

RMSProp was famously introduced by Geoffrey Hinton in a Coursera lecture, not a paper.

optimization rmsprop training

Share this formula

More in Machine Learning & AI

Linear Regression Model

Predicts a continuous value as a weighted sum of input features plus a bias.

Gradient Descent Update

Iteratively moves parameters in the direction that most reduces the loss.

ReLU Activation

Rectified Linear Unit: outputs the input if positive, else zero.

Leaky ReLU

A ReLU variant that lets a small gradient flow for negative inputs.

Tanh Activation

Squashes input to the range (-1, 1); zero-centred activation.

Softmax

Turns a vector of scores into a probability distribution that sums to 1.