ReLU Activation
Rectified Linear Unit: outputs the input if positive, else zero.
Formula
Variables
Example
f(-3)=0, f(5)=5
How it works
ReLU (Rectified Linear Unit) keeps positive input values unchanged and turns negative values into 0, so it acts like a simple gate. This makes neural networks easier to train because it is computationally cheap and avoids the saturation problem seen in some older activation functions, where gradients become very small. In practice, it helps a model learn useful patterns while still being easy to optimize with gradient-based methods.
When to use it
Use ReLU in hidden layers of many neural networks, especially for image, text, and tabular models. It is a common default choice when you want a simple activation that preserves strong positive signals and suppresses negative ones.
Worked example, step by step
Step 1: Start with the input x = 2.7 Step 2: Apply the ReLU rule f(x) = max(0, x) Step 3: Compare the two values 0 and 2.7 Step 4: The larger value is 2.7, so f(2.7) = 2.7 Final answer: the ReLU output is 2.7
Common mistakes to avoid
- Forgetting that all negative inputs become 0
- Using ReLU on the output layer when a different activation is needed
- Thinking ReLU changes positive values instead of passing them through unchanged
Did You Know?
ReLU is the most widely used activation function in deep learning.
Share this formula
More in Data Science & ML
View allMean Squared Error (MSE)
BasicAverage of squared differences between predicted and actual values.
Root Mean Squared Error (RMSE)
BasicSquare root of MSE, in the same units as the target variable.
Mean Absolute Error (MAE)
BasicAverage of absolute differences between predictions and actual values.
Coefficient of Determination (R²)
IntermediateProportion of variance in the target explained by the model.
Classification Accuracy
BasicFraction of predictions that are correct.
Precision
IntermediateOf all positive predictions, the fraction that were correct.