Lesson 1 of 6
Surprise and entropy
Rare events carry more information when they occur. Entropy is the average surprise before we see the outcome.
Core lesson · about 5 minutes · 3 guided parts
Being told the sun rose is unsurprising; being told it snowed on a hot day changes what you know much more.
Your learning route
Build the idea one piece at a time
- Surprisal uses a logarithm
- Entropy is expected surprise
- Use entropy to plan a guessing strategy
Surprisal uses a logarithm
An event with probability 1 gives no news, while smaller probabilities give more bits of news.
Self-information is I(x)=−log₂p(x). The minus sign makes it nonnegative because log p≤0.
Independent surprises add because log of a product is a sum.
Base 2 measures bits; natural logarithms measure nats. Always state the base.
Surprisal answers a question after an outcome occurs: how unexpected was this particular result under the model? An event with probability 1/2 carries 1 bit, 1/4 carries 2 bits, and 1/8 carries 3 bits.
Each additional bit corresponds to halving probability. That is why bits match yes-or-no questions: identifying one of eight equally likely objects takes three balanced binary answers.
The logarithm makes independent evidence additive. If two independent events have probabilities 1/4 and 1/8, their joint probability is 1/32 and surprise is 2+3=5 bits.
Surprise is model-relative. Snow may be unsurprising in the Himalayas and highly surprising in a warm coastal city, because the probability supplied by the two models differs.
I(x)=−log₂ p(x)Worked example
How much information is carried by an event with probability 1/8?
- Use I=−log₂(1/8).
- Since 1/8=2⁻³, log₂(1/8)=−3.
- Negating gives 3 bits.
Answer: 3 bits
Entropy is expected surprise
A certain variable has zero uncertainty; a balanced distribution spreads uncertainty most evenly.
For discrete X, H(X)=−Σp(x)log₂p(x), with 0 log 0 treated as 0.
Among m possible outcomes, entropy is at most log₂m and reaches it for the uniform distribution.
Entropy depends on the whole probability distribution, not the numerical distance between labels.
Entropy averages each outcome's surprisal using its own probability. A common outcome contributes often but carries little surprise; a rare outcome carries much surprise but contributes rarely.
A certain coin with probabilities (1,0) has entropy 0 because its result is known. A fair coin has entropy 1 bit because one balanced yes-or-no answer is needed on average.
Spreading a fixed total probability more evenly increases uncertainty. For four equally likely outcomes entropy is log2(4)=2 bits; concentrating most mass on one outcome makes guessing easier and lowers entropy.
Compression software estimates symbol frequencies, gives short codes to common symbols and long codes to rare ones, and approaches the entropy limit over long messages. Entropy matters because it predicts the average number of bits required before a code is designed.
H(X) = −Σ over x [p(x) log₂ p(x)]Worked example
Find the entropy of a fair coin.
- Both outcomes have probability 1/2.
- Each surprisal is −log₂(1/2)=1 bit.
- Average: (1/2)(1)+(1/2)(1)=1 bit.
Answer: 1 bit
Practical deep dive for this lesson
Use entropy to plan a guessing strategy
A good yes-or-no question splits the remaining probability as evenly as possible because a balanced answer carries the greatest expected information.
Suppose four hidden cards have probabilities 1/2, 1/4, 1/8, and 1/8. Asking 'is it the first card?' makes yes and no each have probability one half, yielding one full bit for the first answer.
After a no, the remaining conditional probabilities are 1/2, 1/4, and 1/4. Another balanced question can again divide the remaining mass, producing shorter paths for common cards and longer paths for rare cards.
This is the same principle used in prefix coding and decision trees. It is useful because balanced splits reduce expected remaining uncertainty, though practical questions may have costs or restrictions not captured by entropy alone.
expected information gain = H(before) − Σ over answers [P(answer) × H(after answer)]Worked example
A clue reduces uncertainty about a hidden object from 3 bits to an average of 1.2 bits. How much information does the clue provide?
- Write information gain as entropy before minus conditional entropy after.
- Substitute 3−1.2.
- Compute the difference 1.8 bits.
- Interpret it as the average number of yes-or-no bits saved by receiving the clue.
Answer: 1.8 bits
Check your picture
Among all distributions on m discrete outcomes with no other constraints, which has maximum entropy and what is that entropy?
Reveal answer
The uniform distribution; log₂(m) bits
This lesson versus the whole chapter
Your topic-specific practical example is above.
The final lesson collects five longer chapter-wide workflows for machine learning, LLMs, trading, physics, and everyday decisions.
Formula shelf
Symbols with meaning
I(x)=−log p(x)Information carried by observing x.
H(X)=−Σp(x)log p(x)Average uncertainty before observing X.
h(X)=−∫f(x)log f(x)dxContinuous spread measure whose value depends on units.
H(X)≤log₂mOn a fixed set of m outcomes, the uniform distribution has the greatest entropy.
H(X,Y)=H(X)+H(Y|X)Describe a pair in conditional stages.
I(X;Y)=H(X)−H(X|Y)Average uncertainty about X removed by Y.
H(p,q)=−Σp log qExpected log loss when q predicts data from p.
DKL(p ∥ q)=Σ over outcomes [p log(p ÷ q)]Extra log loss from distribution mismatch.
C=maximum over input distributions p(x) of I(X;Y)Highest asymptotically reliable information rate.
PPL=exp(mean NLL)Exponentiated average token surprise.
Before moving on
You are ready when you can…
- Compute discrete entropy and explain why continuous differential entropy depends on units
- Identify maximum-entropy distributions only after support and constraints are stated
- Use conditional entropy and mutual information
- Relate cross-entropy to entropy and KL
- Explain source coding, capacity, and data processing
- Interpret log loss and perplexity with their limitations