Machine Learning & AI

Intermediate

Exponential LR Decay

Gradually lowers the learning rate as training progresses.

Formula

\alpha_t = \alpha_0\, e^{-k t}

Variables

\alpha_0Initial rate

kDecay rate

tStep/epoch

Example

alpha0=0.1, k=0.1, t=10: alpha=0.037

Did You Know?

Decaying the learning rate lets a model take big early steps then fine-tune near the minimum.

optimization scheduling training

Share this formula

More in Machine Learning & AI

Linear Regression Model

Predicts a continuous value as a weighted sum of input features plus a bias.

Gradient Descent Update

Iteratively moves parameters in the direction that most reduces the loss.

ReLU Activation

Rectified Linear Unit: outputs the input if positive, else zero.

Leaky ReLU

A ReLU variant that lets a small gradient flow for negative inputs.

Tanh Activation

Squashes input to the range (-1, 1); zero-centred activation.

Softmax

Turns a vector of scores into a probability distribution that sums to 1.