Machine Learning & AI
Intermediate

Adam Optimizer

Adaptive optimizer combining momentum and per-parameter learning rates.

Formula

θ:=θ−α m^tv^t+ϵ\theta := \theta - \alpha\,\dfrac{\hat{m}_t}{\sqrt{\hat{v}_t}+\epsilon}

Variables

\hat{m}_tBias-corrected 1st moment
\hat{v}_t2nd moment
\epsilonSmall constant

Example

Default alpha=0.001 works for most nets

Did You Know?

Adam (2015) is the default optimizer for most deep-learning models today.

optimizationadamtraining

Share this formula

More in Machine Learning & AI

View all

Linear Regression Model

Basic

Predicts a continuous value as a weighted sum of input features plus a bias.

Gradient Descent Update

Basic

Iteratively moves parameters in the direction that most reduces the loss.

ReLU Activation

Basic

Rectified Linear Unit: outputs the input if positive, else zero.

Leaky ReLU

Basic

A ReLU variant that lets a small gradient flow for negative inputs.

Tanh Activation

Basic

Squashes input to the range (-1, 1); zero-centred activation.

Softmax

Basic

Turns a vector of scores into a probability distribution that sums to 1.