Information Gain
Reduction in entropy achieved by splitting a node on a feature.
Formula
Variables
Example
Split drops H from 1.0 to 0.5: IG = 0.5
How it works
Information gain measures how much uncertainty about the class label is removed when a dataset is split by a feature. It starts with the parent node’s entropy and subtracts the average entropy of the child nodes, weighted by how many samples each child contains, so a split is rewarded when it creates purer groups. The formula comes from comparing the “before” uncertainty to the expected “after” uncertainty.
When to use it
Use information gain when building decision trees for classification tasks and you need to decide which feature or threshold gives the best split. It is especially useful in problems like spam detection, medical diagnosis, and customer segmentation where reducing label uncertainty matters.
Worked example, step by step
Suppose a node has 10 samples with parent entropy H(S) = 0.97. A candidate split creates two child subsets: - S1 has 4 samples with entropy H(S1) = 0.25 - S2 has 6 samples with entropy H(S2) = 0.81 Compute the weighted child entropy: (|S1|/|S|)H(S1) + (|S2|/|S|)H(S2) = (4/10)(0.25) + (6/10)(0.81) = 0.10 + 0.486 = 0.586 Now subtract from the parent entropy: IG = H(S) - weighted child entropy IG = 0.97 - 0.586 IG = 0.384 So the information gain is 0.384 bits (or entropy units), meaning the split reduces uncertainty by that amount.
Common mistakes to avoid
- Forgetting to weight each child entropy by its subset size
- Using class probabilities from the whole parent node instead of each child node
- Assuming a larger number of splits always means higher information gain
Did You Know?
Decision trees greedily pick the split with the highest information gain at each step.
Share this formula
More in Machine Learning & AI
View allLinear Regression Model
BasicPredicts a continuous value as a weighted sum of input features plus a bias.
Gradient Descent Update
BasicIteratively moves parameters in the direction that most reduces the loss.
ReLU Activation
BasicRectified Linear Unit: outputs the input if positive, else zero.
Leaky ReLU
BasicA ReLU variant that lets a small gradient flow for negative inputs.
Tanh Activation
BasicSquashes input to the range (-1, 1); zero-centred activation.
Softmax
BasicTurns a vector of scores into a probability distribution that sums to 1.