Exponential LR Decay

Exponential learning-rate decay reduces the step size by a constant multiplicative factor over time, so the rate drops quickly at first and then more slowly. This is useful because large updates early in training help the model move toward a good region of the loss surface, while smaller updates later help fine-tune the weights without overshooting. The formula alpha_t = alpha_0 e^{-kt} encodes that the learning rate shrinks smoothly with training progress, with k controlling how fast the shrinkage happens.

Use exponential decay when you want a predictable, smooth reduction in learning rate during training, especially for neural networks and other gradient-based models. It is common when training needs a strong initial learning rate for fast progress but a smaller later rate for stable convergence.

• Using t as wall-clock time instead of training step/epoch count
• Forgetting that k must be positive for decay, not growth
• Mixing up e^{-kt} with 1 - kt, which is a different kind of schedule