Layer Normalization
Normalizes across features within a single sample; standard in Transformers.
Formula
Variables
Example
Works per-sample, unlike batch norm
How it works
Layer normalization takes the activations from one individual sample, computes their mean and variance across the feature dimension, and rescales them so they have a more controlled spread. The small constant b5 prevents division by zero and keeps the denominator stable when the variance is tiny. The learned parameters b3 and b2 then restore flexibility by letting the model reweight and shift each feature after normalization, so the network can still represent whatever scale it needs.
When to use it
Use this when each example should be normalized independently, especially in sequence models, Transformers, and other architectures where batch sizes may be small or vary a lot. It is also useful when batch statistics would be noisy or awkward to compute, such as online inference or variable-length inputs.
Worked example, step by step
Suppose a sample has four feature values: x = [2, 4, 6, 8] Step 1: Compute the mean across features. b5 = (2 + 4 + 6 + 8) / 4 = 20 / 4 = 5 Step 2: Compute the variance across features. c3^2 = [ (2-5)^2 + (4-5)^2 + (6-5)^2 + (8-5)^2 ] / 4 c3^2 = [9 + 1 + 1 + 9] / 4 = 20 / 4 = 5 Step 3: Normalize each feature using b5 = 0.0001. Denominator = b5c3^2 + b5 = b5(5 + 0.0001) d 2.2361 Normalized values = (x - 5) / 2.2361 d [-1.3416, -0.4472, 0.4472, 1.3416] Step 4: Apply learned scale and shift, with b3 = [1.5, 1.5, 1.5, 1.5] and b2 = [0.5, 0.5, 0.5, 0.5]. 58x_hat = normalized d7 b3 + b2 58x_hat d [-1.5124, -0.1708, 1.1708, 2.5124] Final answer: 58x_hat d [-1.51, -0.17, 1.17, 2.51]
Common mistakes to avoid
- Computing the mean and variance across the batch instead of across features within one sample
- Forgetting the epsilon term, which can make the denominator unstable when variance is very small
- Assuming b3 and b2 are fixed constants instead of learnable parameters
Did You Know?
Layer norm, not batch norm, is used in Transformers because it is independent of batch size.
Share this formula
More in Machine Learning & AI
View allLinear Regression Model
BasicPredicts a continuous value as a weighted sum of input features plus a bias.
Gradient Descent Update
BasicIteratively moves parameters in the direction that most reduces the loss.
ReLU Activation
BasicRectified Linear Unit: outputs the input if positive, else zero.
Leaky ReLU
BasicA ReLU variant that lets a small gradient flow for negative inputs.
Tanh Activation
BasicSquashes input to the range (-1, 1); zero-centred activation.
Softmax
BasicTurns a vector of scores into a probability distribution that sums to 1.