Machine Learning & AI

Intermediate

Information Gain

Reduction in entropy achieved by splitting a node on a feature.

Formula

IG = H(S) - \sum_k \dfrac{|S_k|}{|S|}H(S_k)

Variables

H(S)Parent entropy

S_kChild subsets

Example

Split drops H from 1.0 to 0.5: IG = 0.5

Did You Know?

Decision trees greedily pick the split with the highest information gain at each step.

decision tree split entropy

Share this formula

More in Machine Learning & AI

Linear Regression Model

Predicts a continuous value as a weighted sum of input features plus a bias.

Gradient Descent Update

Iteratively moves parameters in the direction that most reduces the loss.

ReLU Activation

Rectified Linear Unit: outputs the input if positive, else zero.

Leaky ReLU

A ReLU variant that lets a small gradient flow for negative inputs.

Tanh Activation

Squashes input to the range (-1, 1); zero-centred activation.

Softmax

Turns a vector of scores into a probability distribution that sums to 1.