Data Science & ML
Advanced

Shannon Entropy

Measures the uncertainty or information content of a distribution.

Formula

H=−∑i=1npilog⁡2piH = -\sum_{i=1}^{n} p_i \log_2 p_i

Variables

p_iProbability of outcome i
nOutcomes

Example

Fair coin: H = 1 bit

Did You Know?

Entropy is used to choose splits in decision trees.

informationdecision-tree