Dropout (Inverted)
Randomly zeroes activations during training to prevent co-adaptation.
Formula
Variables
Example
p=0.5 drops half the units each step
How it works
In inverted dropout, each activation is randomly kept or set to zero during training, and the surviving activations are scaled up by 1/(1-p) so that their expected value stays the same as without dropout. This keeps the average magnitude of signals consistent between training and inference, which is why no special rescaling is needed at test time. The random mask m changes on each forward pass, forcing the network to learn representations that do not rely too heavily on any single unit.
When to use it
Use inverted dropout when training neural networks that are prone to overfitting, especially fully connected layers or large feature-heavy models. It is most useful when you want a simple, built-in regularizer that makes the model more robust without changing the inference-time computation.
Worked example, step by step
Suppose the activation vector is x = [2.0, 0.5, 4.0, 1.5] and the drop probability is p = 0.25. A possible random mask is m = [1, 0, 1, 1], meaning the second unit is dropped. Compute the scaling factor: 1/(1-p) = 1/(1-0.25) = 1/0.75 = 4/3. Multiply element-wise: x ⊙ m = [2.0, 0, 4.0, 1.5]. Scale the kept activations: \tilde{x} = [2.0, 0, 4.0, 1.5] × 4/3 = [2.6667, 0, 5.3333, 2.0]. Final output: \tilde{x} ≈ [2.67, 0, 5.33, 2.00].
Common mistakes to avoid
- Using dropout scaling at test time instead of during training
- Forgetting that the mask is random and changes every training pass
- Using p as the keep probability instead of the drop probability
Did You Know?
Dropout mimics training an ensemble of many thinner networks that share weights.
Share this formula
More in Machine Learning & AI
View allLinear Regression Model
BasicPredicts a continuous value as a weighted sum of input features plus a bias.
Gradient Descent Update
BasicIteratively moves parameters in the direction that most reduces the loss.
ReLU Activation
BasicRectified Linear Unit: outputs the input if positive, else zero.
Leaky ReLU
BasicA ReLU variant that lets a small gradient flow for negative inputs.
Tanh Activation
BasicSquashes input to the range (-1, 1); zero-centred activation.
Softmax
BasicTurns a vector of scores into a probability distribution that sums to 1.