ReLU Activation
Rectified Linear Unit: outputs the input if positive, else zero.
Formula
Variables
Example
f(-3)=0, f(5)=5
How it works
The ReLU (Rectified Linear Unit) keeps positive inputs unchanged and turns negative inputs into zero. This creates a simple threshold behavior: small or negative signals are suppressed, while stronger positive signals pass through unchanged. It is widely used in neural networks because it is very easy to compute and helps models learn efficiently by avoiding some of the training problems caused by older activation functions.
When to use it
Use ReLU in hidden layers of neural networks when you want a fast, simple activation that preserves positive signals and removes negative ones. It is especially common in deep learning for image recognition, classification, and many other pattern-learning tasks.
Worked example, step by step
Step 1: Start with the input value x = -1.5 Step 2: Apply the formula f(x) = max(0, x) Step 3: Compare 0 and -1.5 Step 4: The larger value is 0, so f(-1.5) = 0 Step 5: Try a positive input, x = 4.2 Step 6: Apply the formula f(x) = max(0, x) Step 7: Compare 0 and 4.2 Step 8: The larger value is 4.2, so f(4.2) = 4.2
Common mistakes to avoid
- Thinking ReLU changes positive numbers in some way; it leaves them exactly the same
- Forgetting that all negative inputs become 0, not a negative output
- Using ReLU as the output layer in every problem without checking whether the task needs probabilities or unrestricted values
Did You Know?
ReLU is astonishingly simple yet enabled the deep-learning boom by avoiding vanishing gradients.
Share this formula
More in Machine Learning & AI
View allLinear Regression Model
BasicPredicts a continuous value as a weighted sum of input features plus a bias.
Gradient Descent Update
BasicIteratively moves parameters in the direction that most reduces the loss.
Leaky ReLU
BasicA ReLU variant that lets a small gradient flow for negative inputs.
Tanh Activation
BasicSquashes input to the range (-1, 1); zero-centred activation.
Softmax
BasicTurns a vector of scores into a probability distribution that sums to 1.
Binary Cross-Entropy
BasicLoss for binary classification comparing predicted probability to the true label.