Leaky ReLU
A ReLU variant that lets a small gradient flow for negative inputs.
Formula
Variables
Example
alpha=0.01, x=-2: f=-0.02
How it works
Leaky ReLU is an activation function used in neural networks that keeps positive inputs unchanged but gives negative inputs a small, nonzero slope. This matters because plain ReLU outputs 0 for all negative values, which can stop learning for those neurons; the small slope in Leaky ReLU lets some gradient still pass during training. The parameter α controls how much signal is preserved on the negative side, so the function is mostly linear but less likely to "die" on negative inputs.
When to use it
Use Leaky ReLU in hidden layers when you want the speed and simplicity of ReLU but need to reduce the risk of dead neurons. It is common in deep feedforward networks, CNNs, and other models where many activations may otherwise become stuck at zero.
Worked example, step by step
Given α = 0.01 and x = -5 Step 1: Compute αx = 0.01 × (-5) = -0.05 Step 2: Compute x = -5 Step 3: Take the larger value: max(-0.05, -5) = -0.05 Final answer: f(x) = -0.05
Common mistakes to avoid
- Using the wrong branch for negative inputs and outputting 0 instead of αx
- Choosing an α value that is too large, which makes the function behave too much like a normal linear unit
- Forgetting that the max compares αx and x, not α and x separately
Did You Know?
Leaky ReLU fixes the "dying ReLU" problem where neurons get stuck outputting zero.
Share this formula
More in Machine Learning & AI
View allLinear Regression Model
BasicPredicts a continuous value as a weighted sum of input features plus a bias.
Gradient Descent Update
BasicIteratively moves parameters in the direction that most reduces the loss.
ReLU Activation
BasicRectified Linear Unit: outputs the input if positive, else zero.
Tanh Activation
BasicSquashes input to the range (-1, 1); zero-centred activation.
Softmax
BasicTurns a vector of scores into a probability distribution that sums to 1.
Binary Cross-Entropy
BasicLoss for binary classification comparing predicted probability to the true label.