Entropy (Information)
Measures impurity/uncertainty of a set; used to split decision trees.
Formula
Variables
Example
Even 2-class split: H = 1 bit
How it works
Information entropy measures how mixed a dataset is by turning class proportions into a weighted average of surprise. The logarithm means rare classes contribute more uncertainty than common ones, and the negative sign makes the total nonnegative because each \(\log_2(p_i)\) is negative when \(0<p_i<1\). A set is most uncertain when its classes are evenly balanced, and less uncertain when one class dominates.
When to use it
Use this when you need a numeric measure of class impurity, especially in decision-tree algorithms like ID3, C4.5, or related split-selection methods. It is also useful anywhere you want to compare how mixed or predictable labeled data is before and after a partition.
Worked example, step by step
Suppose a dataset has 10 items: 5 of class A, 3 of class B, and 2 of class C. Step 1: Compute class proportions: p_A = 5/10 = 0.5, p_B = 3/10 = 0.3, p_C = 2/10 = 0.2 Step 2: Substitute into the entropy formula: H(S) = -[0.5 log_2(0.5) + 0.3 log_2(0.3) + 0.2 log_2(0.2)] Step 3: Evaluate the logs: log_2(0.5) = -1 log_2(0.3) ≈ -1.737 log_2(0.2) ≈ -2.322 Step 4: Multiply each term: 0.5(-1) = -0.500 0.3(-1.737) ≈ -0.521 0.2(-2.322) ≈ -0.464 Step 5: Add and negate: H(S) = -[-0.500 - 0.521 - 0.464] = 1.485 bits Final answer: H(S) ≈ 1.49 bits
Common mistakes to avoid
- Using raw class counts instead of proportions that sum to 1
- Forgetting the negative sign, which makes the result appear negative
- Using natural log without adjusting the unit, instead of log base 2 for bits
Did You Know?
Claude Shannon introduced entropy in 1948, founding information theory itself.
Share this formula
More in Machine Learning & AI
View allLinear Regression Model
BasicPredicts a continuous value as a weighted sum of input features plus a bias.
Gradient Descent Update
BasicIteratively moves parameters in the direction that most reduces the loss.
ReLU Activation
BasicRectified Linear Unit: outputs the input if positive, else zero.
Leaky ReLU
BasicA ReLU variant that lets a small gradient flow for negative inputs.
Tanh Activation
BasicSquashes input to the range (-1, 1); zero-centred activation.
Softmax
BasicTurns a vector of scores into a probability distribution that sums to 1.