L2 Regularization (Ridge)
Adds a penalty on squared weights to reduce overfitting.
Formula
Variables
Example
Large lambda shrinks all weights toward zero
How it works
L2 regularization modifies the usual prediction error by adding a cost for large weights, so the model is encouraged to fit the data while keeping coefficients small. Because large weights can make a model overly sensitive to noise and to small changes in the training set, this penalty usually improves generalization. The squared form is smooth and differentiable, which makes it easy to optimize with gradient-based methods.
When to use it
Use it when a model is overfitting, especially in linear regression, logistic regression, and neural networks with many parameters. It is also helpful when many input features are correlated and you want a stable model with smaller, less extreme weights.
Worked example, step by step
Suppose a model has MSE = 4.8, weights w1 = 2, w2 = -1, and regularization strength λ = 0.5. Step 1: Square the weights: w1^2 = 2^2 = 4, w2^2 = (-1)^2 = 1 Step 2: Add the squared weights: Σ wj^2 = 4 + 1 = 5 Step 3: Multiply by λ: λΣ wj^2 = 0.5 × 5 = 2.5 Step 4: Add this penalty to the MSE: J = 4.8 + 2.5 = 7.3 Final answer: J = 7.3
Common mistakes to avoid
- Forgetting to square the weights before summing them
- Assuming a larger λ always improves performance; too much regularization can cause underfitting
- Including the bias/intercept term in the penalty when the model definition excludes it
Did You Know?
L2 shrinks weights smoothly toward zero but rarely makes them exactly zero.
Share this formula
More in Machine Learning & AI
View allLinear Regression Model
BasicPredicts a continuous value as a weighted sum of input features plus a bias.
Gradient Descent Update
BasicIteratively moves parameters in the direction that most reduces the loss.
ReLU Activation
BasicRectified Linear Unit: outputs the input if positive, else zero.
Leaky ReLU
BasicA ReLU variant that lets a small gradient flow for negative inputs.
Tanh Activation
BasicSquashes input to the range (-1, 1); zero-centred activation.
Softmax
BasicTurns a vector of scores into a probability distribution that sums to 1.