L1 Regularization (Lasso)
Penalizes absolute weights, driving many to exactly zero for feature selection.
Formula
Variables
Example
Lasso can zero out irrelevant features
How it works
L1 regularization adds a penalty equal to the sum of the absolute values of the model weights, so the optimizer must balance fit quality against keeping coefficients large. Because absolute-value penalties grow linearly, they strongly encourage some weights to shrink all the way to zero, which makes the model simpler and often easier to interpret. This is especially useful when many input features may be redundant or weakly relevant.
When to use it
Use Lasso when you expect only a small subset of features to matter and you want automatic feature selection built into training. It is common in high-dimensional problems such as text data, genomics, or any dataset with many correlated or noisy predictors.
Worked example, step by step
Suppose a linear model has mean squared error MSE = 12.0, weights w1 = 3, w2 = -1, and w3 = 0.5, with λ = 2. Step 1: Compute the absolute values: |w1| = 3, |w2| = 1, |w3| = 0.5 Step 2: Sum them: 3 + 1 + 0.5 = 4.5 Step 3: Multiply by λ: 2 × 4.5 = 9.0 Step 4: Add the penalty to the MSE: J = 12.0 + 9.0 = 21.0 Final answer: J = 21.0
Common mistakes to avoid
- Using the raw weights instead of their absolute values
- Forgetting that the penalty is added to the loss, not subtracted
- Assuming L1 only makes weights smaller rather than sometimes setting them exactly to zero
Did You Know?
L1 produces sparse models — effectively doing automatic feature selection.
Share this formula
More in Machine Learning & AI
View allLinear Regression Model
BasicPredicts a continuous value as a weighted sum of input features plus a bias.
Gradient Descent Update
BasicIteratively moves parameters in the direction that most reduces the loss.
ReLU Activation
BasicRectified Linear Unit: outputs the input if positive, else zero.
Leaky ReLU
BasicA ReLU variant that lets a small gradient flow for negative inputs.
Tanh Activation
BasicSquashes input to the range (-1, 1); zero-centred activation.
Softmax
BasicTurns a vector of scores into a probability distribution that sums to 1.