Mathematics for AI 02 | Why Information Is Measured in Bits

Mathematics for AI 02 Why does a fair coin give one bit of information, while one of eight equally likely cards gives three? Follow the choices, the binary tree, and the logarithm to see how probability becomes a measure of information. In episode 2 of Mathematics for AI, Emma, Aoi, and Norm explain self-information (surprisal), why the unit uses base two, and why logarithms let information from successive steps add up. Then we use a small language-model example to distinguish scoring an observed token during training from choosing the next token during generation. 00:00 A measure for surprise 00:57 One balanced choice, one bit 01:59 Why a logarithm? 05:28 The model matters 06:21 Scoring an AI prediction 09:21 One outcome, then an average The central definition is I = −log₂(p), measured in bits. Here, surprise is a mathematical quantity under a specified probability model, not a measurement of emotion, importance, or truth. The language-model probabilities are illustrative examples, not measurements of a particular AI system. Series: Mathematics for AI 01 — Bayes' rule: updating beliefs 02 — Information, surprise, and bits (this episode) 03 — Entropy: measuring uncertainty (next) 04 — Cross-entropy and prediction loss 05 — Probability calibration 06 — Gradient descent and learning Future topics and order are provisional. References and further reading: Claude E. Shannon, A Mathematical Theory of Communication (1948): https://onlinelibrary.wiley.com/doi/pdf/10.1002/j.1538-7305.1948.tb01338.x Stanford EE276, Information Theory lecture notes: https://web.stanford.edu/class/ee276/files/previous_notes/lecture_2.pdf Hugging Face Transformers, Perplexity: https://huggingface.co/docs/transformers/perplexity Hugging Face Transformers, Generation strategies: https://huggingface.co/docs/transformers/generation_strategies Original diagrams and animation: Curiosity, Clearly. Fictional illustrated presenters and synthetic character voices. If the explanation helped, like the video and subscribe. What everyday prediction would you measure in bits? Tell us in the comments. #InformationTheory #MathematicsForAI #Bits Series playlist: https://www.youtube.com/playlist?list=PLGi5Uwmf0LpY Previous episode: https://youtu.be/kS9gsyHW0yk