Information Theory

Entropy, mutual information, and the coding view of learning. The measure of how much a message, a feature, or a model actually tells you.

7 items

Encyclopedia (3)

Entropy

A measure of the uncertainty in a random variable, equal to the average number of bits needed to encode its outcome.

Cross-Entropy

A measure of the difference between two probability distributions, used as the standard loss function for classification.

Perplexity

The exponential of a model’s average cross entropy, read as the number of equally likely options it is effectively choosing between at each step.

Articles (1)

Measuring uncertainty in bits: Shannon entropy and why the logarithm is base 2, information gain worked on a split, and how cross-entropy and KL divergence relate to entropy and to the loss functions used to train classifiers.

Tools (1)

Entropy and information calculator

Edit two distributions side by side and read off entropy, cross entropy, and KL divergence in bits, with the identity H(p, q) = H(p) + KL(p ‖ q) held on screen.

Research (1)

A Mathematical Theory of Communication

Defines information quantitatively, introduces entropy as the measure of a source’s uncertainty, and proves limits on lossless compression and on reliable transmission over a noisy channel.

Projects (1)

Bayesian Inference Playground

Prior to posterior updating with conjugate families, alongside an entropy and KL-divergence explorer that measures what each observation actually told you.