A Mathematical Theory of Communication

Summary

Defines information quantitatively, introduces entropy as the measure of a source’s uncertainty, and proves limits on lossless compression and on reliable transmission over a noisy channel.

Why it matters

It founded information theory outright. Entropy, defined here, is now the unit in which uncertainty is measured across statistics and machine learning: it is what information gain reduces when a decision tree splits, and what cross-entropy generalizes when a model’s predicted distribution is scored against reality.