Computer Science
Advanced

Shannon Entropy

Average information content of a random source (bits).

Formula

H=−∑ipilog⁡2piH = -\sum_i p_i \log_2 p_i

Variables

p_iSymbol probabilities

Example

A fair coin carries H = −(½·log₂½ + ½·log₂½) = 1 bit of information per flip.

Did You Know?

Claude Shannon founded information theory in 1948, giving us the bit as the fundamental unit of information.

entropyinformationbits