Machine Learning & AI

Intermediate

Entropy (Information)

Measures impurity/uncertainty of a set; used to split decision trees.

Formula

H(S) = -\sum_i p_i \log_2 p_i

Variables

SDataset

p_iProportion of class i

Example

Even 2-class split: H = 1 bit

Did You Know?

Claude Shannon introduced entropy in 1948, founding information theory itself.

decision tree information entropy

Share this formula

More in Machine Learning & AI

Linear Regression Model

Predicts a continuous value as a weighted sum of input features plus a bias.

Gradient Descent Update

Iteratively moves parameters in the direction that most reduces the loss.

ReLU Activation

Rectified Linear Unit: outputs the input if positive, else zero.

Leaky ReLU

A ReLU variant that lets a small gradient flow for negative inputs.

Tanh Activation

Squashes input to the range (-1, 1); zero-centred activation.

Softmax

Turns a vector of scores into a probability distribution that sums to 1.