#kl-divergence · Kudos AI

14 September 20264 min readProbability Foundations

Which Wrong Distribution Do You Want?

One bimodal target, one Gaussian, and two directions of the same divergence. Minimising KL(P||Q) puts the Gaussian across both modes with almost no mass where the target actually lives; minimising KL(Q||P) puts it on one mode at a value of 0.6931 nats, which is ln 2 to four decimals and not a coincidence. Each fit is judged catastrophic by the other objective, 2.0976 against 15.2799.

Machine LearningMathematics

29 August 202610 min readProbability Foundations

Entropy and Information

Measuring uncertainty in bits: Shannon entropy and why the logarithm is base 2, information gain worked on a split, and how cross-entropy and KL divergence relate to entropy and to the loss functions used to train classifiers.

Information TheoryProbabilityMathematics