Machine Learning & AI
Advanced

Layer Normalization

Normalizes across features within a single sample; standard in Transformers.

Formula

x^=x−μσ2+ϵ γ+β\hat{x}=\dfrac{x-\mu}{\sqrt{\sigma^2+\epsilon}}\,\gamma+\beta

Variables

\muFeature mean
\sigma^2Feature variance
\gamma,\betaLearned scale/shift

Example

Works per-sample, unlike batch norm

Did You Know?

Layer norm, not batch norm, is used in Transformers because it is independent of batch size.

transformernormalizationdeep learning

Share this formula

More in Machine Learning & AI

View all

Linear Regression Model

Basic

Predicts a continuous value as a weighted sum of input features plus a bias.

Gradient Descent Update

Basic

Iteratively moves parameters in the direction that most reduces the loss.

ReLU Activation

Basic

Rectified Linear Unit: outputs the input if positive, else zero.

Leaky ReLU

Basic

A ReLU variant that lets a small gradient flow for negative inputs.

Tanh Activation

Basic

Squashes input to the range (-1, 1); zero-centred activation.

Softmax

Basic

Turns a vector of scores into a probability distribution that sums to 1.