Bias-Variance Decomposition
Total expected error decomposes into bias, variance, and irreducible noise.
Formula
Variables
Example
High bias = underfit, high variance = overfit
How it works
The bias-variance decomposition explains why a model’s expected prediction error can be split into three parts: systematic error from using the wrong kind of model or overly strong assumptions (bias), sensitivity to the particular training sample (variance), and randomness in the data that no model can remove (c3^2). It works because, for squared error, the expected prediction can be algebraically expanded into the distance from the true target, plus the variability around that expectation, plus the unavoidable noise in the observed outcomes. This makes it a useful diagnostic framework for understanding whether improving a model should focus on making it more flexible, making it more stable, or accepting the noise floor.
When to use it
Use this when analyzing supervised learning models under squared-error loss, especially to reason about generalization error and the underfitting/overfitting tradeoff. It is especially helpful in model selection, ensemble design, and comparing simple versus complex learners on the same dataset.
Worked example, step by step
Suppose a regression model has estimated components: Bias = 3, Variance = 4, and irreducible noise c3^2 = 5. Step 1: Square the bias term: Bias^2 = 3^2 = 9 Step 2: Keep the variance term as is: Variance = 4 Step 3: Add the irreducible noise: c3^2 = 5 Step 4: Combine them: Error = 9 + 4 + 5 = 18 Final answer: the expected total error is 18.
Common mistakes to avoid
- Forgetting that the bias term is squared, not just added linearly
- Treating irreducible noise as something a better model can eliminate
- Assuming a low-bias model must also have low total error, even if its variance is large
Did You Know?
The bias-variance tradeoff guides model complexity choices.
Share this formula
More in Data Science & ML
View allMean Squared Error (MSE)
BasicAverage of squared differences between predicted and actual values.
Root Mean Squared Error (RMSE)
BasicSquare root of MSE, in the same units as the target variable.
Mean Absolute Error (MAE)
BasicAverage of absolute differences between predictions and actual values.
Coefficient of Determination (R²)
IntermediateProportion of variance in the target explained by the model.
Classification Accuracy
BasicFraction of predictions that are correct.
Precision
IntermediateOf all positive predictions, the fraction that were correct.