Shannon Entropy
Measures the uncertainty or information content of a distribution.
Formula
Variables
Example
Fair coin: H = 1 bit
How it works
Shannon entropy quantifies how unpredictable a random variable is by averaging the self-information of its possible outcomes, where rarer outcomes contribute more surprise. The negative sign makes the result positive, since probabilities are between 0 and 1 and their base-2 logarithms are negative. It is maximized when outcomes are equally likely and drops toward 0 when one outcome dominates, which matches the intuition that a highly skewed distribution is easier to guess.
When to use it
Use this when you want a principled measure of uncertainty in a categorical distribution, such as in decision trees, feature selection, language modeling, or comparing class imbalance. It is also useful in information theory whenever you need to quantify the average number of bits needed to encode outcomes from a distribution.
Worked example, step by step
Suppose a die is biased so the probabilities are p = [0.5, 0.25, 0.125, 0.125] for four outcomes. H = -\sum p_i \log_2 p_i H = -[0.5\log_2(0.5) + 0.25\log_2(0.25) + 0.125\log_2(0.125) + 0.125\log_2(0.125)] log_2(0.5) = -1, log_2(0.25) = -2, log_2(0.125) = -3 H = -[0.5(-1) + 0.25(-2) + 0.125(-3) + 0.125(-3)] H = -[-0.5 - 0.5 - 0.375 - 0.375] H = 1.75 bits
Common mistakes to avoid
- Forgetting the negative sign, which makes the entropy come out negative
- Using counts instead of probabilities without normalizing them to sum to 1
- Mixing logarithm bases; base 2 gives entropy in bits, while other bases change the units
Did You Know?
Entropy is used to choose splits in decision trees.
Share this formula
More in Data Science & ML
View allMean Squared Error (MSE)
BasicAverage of squared differences between predicted and actual values.
Root Mean Squared Error (RMSE)
BasicSquare root of MSE, in the same units as the target variable.
Mean Absolute Error (MAE)
BasicAverage of absolute differences between predictions and actual values.
Coefficient of Determination (R²)
IntermediateProportion of variance in the target explained by the model.
Classification Accuracy
BasicFraction of predictions that are correct.
Precision
IntermediateOf all positive predictions, the fraction that were correct.