Adam Optimizer

Adam updates parameters by taking a step in the direction suggested by the gradient, but it first smooths that gradient with momentum and then rescales the step using an estimate of how variable each parameter’s recent gradients have been. The first moment term ̂m_t captures the average direction, while the second moment term ̂v_t dampens updates where gradients have been large or noisy, so unstable dimensions do not dominate the learning process. The small ε prevents division by zero and keeps the update numerically stable.

Use Adam when training neural networks or other differentiable models with noisy, sparse, or differently scaled gradients, especially if you want a robust default optimizer without manually tuning a separate learning rate for each parameter. It is commonly used for deep learning tasks such as image, text, and sequence models.

• Forgetting that \hat{m}_t and \hat{v}_t are bias-corrected, especially early in training
• Using \hat{v}_t without the square root, which changes the scale of the update
• Choosing an ε that is too large and unintentionally dominating the denominator