Tagged “mutual-information”

2 articles.

14 September 20263 min readProbability Foundations

The Two Features That Look Like Noise

A variable that determines another with a correlation of exactly 0.0000000000, and a pair of features whose every pairwise mutual information with the target is exactly zero while the two together determine it completely. Univariate screening discards both, and the second case is the one that matters: the features it removes are removed because they matter.

Machine LearningMathematics

11 September 20267 min readInformation Theory

The Bound That Is Actually Reached

Entropy is not a summary of a distribution but a floor that the best code meets to the last decimal, the surcharge for using the wrong distribution is exactly the loss every classifier already minimises, and mutual information puts a hard ceiling on everything downstream of a sensor. Three results, each unusually sharp.

MathematicsMachine Learning