Machine Learning & AI

Basic

Leaky ReLU

A ReLU variant that lets a small gradient flow for negative inputs.

Formula

f(x) = \max(\alpha x, x)

Variables

xInput

\alphaSmall slope (e.g. 0.01)

Example

alpha=0.01, x=-2: f=-0.02

Did You Know?

Leaky ReLU fixes the "dying ReLU" problem where neurons get stuck outputting zero.

activation deep learning

Share this formula

More in Machine Learning & AI

Linear Regression Model

Predicts a continuous value as a weighted sum of input features plus a bias.

Gradient Descent Update

Iteratively moves parameters in the direction that most reduces the loss.

ReLU Activation

Rectified Linear Unit: outputs the input if positive, else zero.

Tanh Activation

Squashes input to the range (-1, 1); zero-centred activation.

Softmax

Turns a vector of scores into a probability distribution that sums to 1.

Binary Cross-Entropy

Loss for binary classification comparing predicted probability to the true label.