Shannon Entropy
Average information content — and the theoretical strength of a random source.
Formula
Variables
Example
Fair coin: H = -2(0.5 log 0.5) = 1 bit
How it works
Shannon entropy measures how unpredictable a source is by averaging the information gained when each symbol appears. The logarithm appears because information should add when independent events are combined, and the minus sign makes the total positive since probabilities are between 0 and 1. A source is most uncertain when its probabilities are spread out evenly, and less uncertain when one outcome dominates.
When to use it
Use this formula when you want to quantify uncertainty, randomness, or information content in a set of possible outcomes. In cryptography and security, it helps assess how strong a password, key, token, or random generator is in principle.
Worked example, step by step
Suppose a message source emits four symbols with probabilities p = {0.5, 0.25, 0.125, 0.125}. H = -\sum p_i \log_2 p_i H = -(0.5\log_2 0.5 + 0.25\log_2 0.25 + 0.125\log_2 0.125 + 0.125\log_2 0.125) log_2 0.5 = -1, log_2 0.25 = -2, log_2 0.125 = -3 H = -(0.5(-1) + 0.25(-2) + 0.125(-3) + 0.125(-3)) H = -(-0.5 - 0.5 - 0.375 - 0.375) H = 1.75 bits So the average information content of one symbol is 1.75 bits.
Common mistakes to avoid
- Forgetting the minus sign, which makes the entropy come out negative
- Using log base 10 or natural log without converting the units to bits
- Including probabilities incorrectly, especially when the probabilities do not sum to 1
Did You Know?
Claude Shannon coined the term 'bit' and founded information theory in a single 1948 paper.
Share this formula
More in Cryptography & Security
View allModular Exponentiation
IntermediateCore operation of RSA encryption — raise a message to a power, modulo n.
RSA Decryption
AdvancedRecovers the message using the private exponent d.
Euler's Totient
IntermediateCounts integers below n coprime to it, for n = pq — used to make RSA keys.
Password Entropy
IntermediateBits of security from password length and character-set size.
Key Space Size
IntermediateNumber of possible keys for a b-bit key.
Birthday Bound
AdvancedCollisions become likely after about √ of the hash space.