Batch Normalization
Normalizes layer inputs across a mini-batch to speed and stabilize training.
Formula
Variables
Example
Rescales activations to ~zero mean, unit variance
How it works
Batch normalization standardizes each activation using the mean and variance computed from the current mini-batch, so the network sees inputs that are on a more consistent scale during training. The small constant epsilon prevents division by zero and keeps the denominator numerically stable when the batch variance is tiny. This usually makes optimization smoother, reduces sensitivity to weight initialization and learning rate, and can help gradients propagate more reliably through deep networks.
When to use it
Use it in deep neural networks, especially feedforward nets and convolutional models, when training is unstable, slow, or very sensitive to hyperparameters. It is most useful during mini-batch training, where batch statistics are available and the model benefits from normalized intermediate activations.
Worked example, step by step
Suppose a mini-batch of one neuron's activations is x = [6, 8, 10, 12]. Step 1: Compute the batch mean: μ_B = (6 + 8 + 10 + 12) / 4 = 36 / 4 = 9 Step 2: Compute the batch variance: σ_B^2 = [(6−9)^2 + (8−9)^2 + (10−9)^2 + (12−9)^2] / 4 = (9 + 1 + 1 + 9) / 4 = 20 / 4 = 5 Step 3: Choose a stability constant, say ε = 0.001 Step 4: Normalize each value using x̂ = (x − μ_B) / √(σ_B^2 + ε) = (x − 9) / √(5.001) Step 5: √(5.001) ≈ 2.2363, so the normalized values are: (6 − 9) / 2.2363 ≈ −1.342 (8 − 9) / 2.2363 ≈ −0.447 (10 − 9) / 2.2363 ≈ 0.447 (12 − 9) / 2.2363 ≈ 1.342 Final answer: x̂ ≈ [−1.342, −0.447, 0.447, 1.342]
Common mistakes to avoid
- Using the mean and variance from the entire dataset instead of the current mini-batch during training
- Forgetting to add ε, which can cause numerical instability when the batch variance is very small
- Confusing batch normalization with final output scaling; it normalizes intermediate activations, not the target labels
Did You Know?
Batch norm (2015) let networks train far faster and go much deeper.
Share this formula
More in Machine Learning & AI
View allLinear Regression Model
BasicPredicts a continuous value as a weighted sum of input features plus a bias.
Gradient Descent Update
BasicIteratively moves parameters in the direction that most reduces the loss.
ReLU Activation
BasicRectified Linear Unit: outputs the input if positive, else zero.
Leaky ReLU
BasicA ReLU variant that lets a small gradient flow for negative inputs.
Tanh Activation
BasicSquashes input to the range (-1, 1); zero-centred activation.
Softmax
BasicTurns a vector of scores into a probability distribution that sums to 1.