Shannon Entropy
Average information content of a random source (bits).
Formula
Variables
Example
A fair coin carries H = −(½·log₂½ + ½·log₂½) = 1 bit of information per flip.
How it works
Shannon entropy measures how unpredictable a source is by averaging the information contributed by each possible outcome. Rare outcomes contribute more because receiving a surprising event tells you more than receiving something expected; the minus sign makes the total positive since log2(p_i) is negative for probabilities between 0 and 1. The logarithm base 2 means the result is measured in bits, so the entropy tells you the ideal average number of binary yes/no questions needed to identify the outcome.
When to use it
Use this when analyzing uncertainty in messages, symbols, data compression, or probability distributions in machine learning and information theory. It is especially useful for comparing how predictable two sources are or for measuring how much information is gained when one outcome is observed.
Worked example, step by step
Suppose a source emits three symbols with probabilities p(A)=0.5, p(B)=0.25, and p(C)=0.25. H = -(0.5 log2 0.5 + 0.25 log2 0.25 + 0.25 log2 0.25) log2 0.5 = -1, and log2 0.25 = -2 H = -(0.5 × -1 + 0.25 × -2 + 0.25 × -2) H = -(-0.5 - 0.5 - 0.5) H = 1.5 bits So the average information content of one symbol from this source is 1.5 bits.
Common mistakes to avoid
- Forgetting the minus sign, which would make the entropy negative
- Using the wrong logarithm base and getting the answer in nats instead of bits
- Treating entropy as the surprise of one outcome rather than the average over the whole distribution
Did You Know?
Claude Shannon founded information theory in 1948, giving us the bit as the fundamental unit of information.
Share this formula
More in Computer Science
View allMaster Theorem (form)
AdvancedAnalyzes running time of divide-and-conquer algorithms.
Hamming Distance
AdvancedNumber of positions at which two strings differ.
Bits to Represent n
BasicMinimum number of bits to represent n distinct values.
Binary to Decimal
BasicConverts a binary number to decimal.
Nodes in a Full Binary Tree
IntermediateTotal nodes in a perfect binary tree of height h.
Decimal to Binary
BasicConvert a decimal number to binary by dividing by 2 and reading remainders.