EasySurprisalinformation-theory-1
Chapter 15 practice bank
100 questions:
Information
Solve on paper first. Reveal only the final answer when you want to check yourself. There are no worked solutions in this bank by design.
34 easy 33 medium 33 advanced
EasyMaximum entropyinformation-theory-2
Among all distributions on 2 discrete outcomes, which one has maximum entropy, and what is that entropy in bits?
EasyLog lossinformation-theory-3
A model assigns probability 1/2 to the observed class. Find its log loss in bits.
EasyEntropy chain ruleinformation-theory-4
H(X)=2 bits and H(Y|X)=1 bits. Find H(X,Y).
EasyMutual informationinformation-theory-5
H(X)=3 bits and H(X|Y)=1 bits. Find I(X;Y).
EasyPerplexityinformation-theory-6
A model has average cross-entropy 1 bits per token. Find its perplexity.
EasyAverage code lengthinformation-theory-7
Symbol probabilities are 0.5, 0.25, 0.25 and code lengths are 1, 2, 2. Find average length.
EasyKL divergenceinformation-theory-8
In bits, p puts all mass on one outcome while q gives that outcome probability 1/2. Find DKL(p∥q).
EasySurprisalinformation-theory-9
An event has probability 1/4. How many bits of surprisal does it carry?
EasyMaximum entropyinformation-theory-10