Series, deep dives, and landmark-paper reviews across 25+ knowledge domains, written to be read closely.
Start here
New here? Begin with the first article and follow the path stage by stage.
Probability from Zero: The Language of Uncertainty
First article in Probability Foundations
A guided path through the mathematics of machine learning, from first principles to the research frontier. Follow a track top to bottom, or jump straight to the area you need.
01
The mathematics every later article assumes: how to reason about uncertainty, and what it means to estimate an unknown function from a finite, noisy sample.
02
Supervised methods built from the ground up, each derived rather than described, and each worked through on a dataset small enough to check by hand.
03
How networks learn by following gradients, and the attention mechanism that modern language models are built around.
04
Deciding what to do when someone else is deciding too: optimal play against an adversary, and equilibrium when interests only partly conflict.
Prefer to explore by topic or difficulty? Filter the full catalogue below.
One bimodal target, one Gaussian, and two directions of the same divergence. Minimising KL(P||Q) puts the Gaussian across both modes with almost no mass where the target actually lives; minimising KL(Q||P) puts it on one mode at a value of 0.6931 nats, which is ln 2 to four decimals and not a coincidence. Each fit is judged catastrophic by the other objective, 2.0976 against 15.2799.
Take the most likely state on each day and write them down in order, and you have a report the model assigns probability exactly zero: on a four-day machine-monitoring example the day-by-day answer is healthy, healthy, failed, failed, and healthy to failed is a transition that cannot occur. What the two questions actually are, why smoothing and Viterbi answer different ones, and what the 0.411 posterior on the best path means for anyone who has to act on it.
A variable that determines another with a correlation of exactly 0.0000000000, and a pair of features whose every pairwise mutual information with the target is exactly zero while the two together determine it completely. Univariate screening discards both, and the second case is the one that matters: the features it removes are removed because they matter.
Averaged over all 256 functions from three bits to one, a nearest-neighbour learner and a learner built to be wrong on purpose both score exactly 0.500000 off the training set. That is the no free lunch theorem, it is exactly true, and the moment the average is restricted to the six functions that depend on a single bit the two separate to 0.333333 and 0.666667.
The step size you are allowed is fixed by the steepest direction and the number of steps you need is fixed by the flattest, so the cost of gradient descent is their ratio. The same least-squares fit, to the same ten decimal places, takes 1742 steps in one basis, 147 in a rescaled one and exactly 1 in an orthonormal one, and momentum buys back the square root of the ratio rather than the ratio.
The living reward in a grid world is written down once and never discussed, and the optimal policy is a step function of it: eight thresholds between -3 and 0, each flipping exactly one square. The textbook value of -0.04 sits 0.0048 away from the one that decides whether the agent takes the shortcut past the pit, and above -0.0221, when steps are nearly free, the optimal move in one corner is to walk into a wall on purpose.
Allowing sideways moves takes hill climbing on 8 queens from 14.75% of runs solved to 94.55%, which reads like a six-fold improvement and is not one: with random restarts the expected cost of a solution goes from 21.9 steps to 23.1, and counted in moves evaluated it falls by 16%, from 1,547 to 1,298. Simulated annealing solves 98.8% and costs 1,622 evaluations. What changed was mostly the statistic, not the work.
Twelve people, two measurements, and three different first principal components: in millimetres the answer is almost pure height, in metres almost pure weight, and in centimetres an even blend - with the correlation fixed at 0.9500 throughout. What that says about what PCA maximises, why a proportion of variance explained of 99.999% can be a statement about metres rather than about people, and what standardising actually chooses.
Two independent causes and one common effect. Adjust for the effect and the causes acquire a correlation of exactly -1: a regression of A on B recovers a coefficient of +0.0030, and adding the common effect as a control turns it into -1.0000. Selecting a sample does the same thing invisibly, which is why "control for everything you measured" is not a defensible rule.
The textbook confidence interval for a proportion has exact coverage you can compute by summing over the n+1 possible samples, and at n = 30 with p = 0.10 it is 0.8085 rather than 0.95. Coverage does not improve monotonically with n, and in a rare-event setting it can fall to 0.0392. Two one-line alternatives fix it.
A classifier with exactly one real parameter fits all 1,048,576 labellings of twenty points, every time, and predicts a twenty-first at 0.5038 accuracy over twenty thousand trials. Counting parameters measures neither an upper nor a lower bound on what a model class can fit, which is why capacity has to be measured some other way.
A five-nearest-neighbour model scores 0.9983 under random five-fold cross-validation on a random walk, a series whose increments are by construction unpredictable. Evaluated forward in time it scores 0.6559 with an RMSE 12.44 times larger, and loses to carrying the last observed value forward. The split, not the model, produced the first number.
Converting one short formula to conjunctive normal form by distributing gives 1,048,576 clauses and 20,971,520 literals; naming the subformulas gives 61 clauses and 160 literals, a factor of 131,072 in literals, and loses nothing at all - the two have the same number of models, checked exhaustively. The encoding is where a satisfiability problem is won or lost, not the solver.
On the burglary network with both neighbours calling, rejection sampling keeps 183 of 100,000 draws and likelihood weighting keeps all of them at an effective sample size of 396. Both estimates are about 10% off a posterior of 0.284172, and the reason is exactly computable: 252 samples carry 76% of the weight and 99.975% of the squared weight.
A two per cent change in the learning rate separates a converged run from one five orders of magnitude away, a condition number predicts the convergence rate to six decimal places, and stochastic gradient descent with a fixed step never converges at all - it settles into a ball whose radius grows as the square root of the step. Every figure here was computed on a problem whose exact optimum is known.
At a realistic base rate the do-nothing detector wins on accuracy, a ROC of 0.9468 hides an alert queue that is 64% false, distance from the mean scores below chance when anomalies sit at the centre, and twenty anomalies that group together hide each other from the method built to find them.
The gap between the error you measure and the error you will suffer, why picking the best of a thousand identical hypotheses makes it look 0.1149 better than chance, how capacity is counted for infinite model classes, and the theorem that equalises every learner - with the assumption that makes it true.
A treatment that raises recovery by exactly five points in every subgroup while appearing to lower it overall, why more data makes that conclusion more confident rather than more correct, what randomisation buys that adjustment cannot, and the case where controlling for a variable manufactures an association from nothing.
Two series generated from separate random numbers come out significantly related 82.8% of the time, a standard error on dependent data is too narrow by a computable factor of 2.4, and the usual validation split reports a forecaster more than five times better than it is. Three failures, one cause, and the checks that catch each of them.
Two fitted offsets deliver 66% of a recommender’s gain in accuracy before any latent factor is learned, the error is 1.28 times worse for the users who have said least, only 30% of the catalogue reaches anyone’s top ten with no explicit popularity term in the model, and after six rounds of self-selected data the system is 1.14 times worse exactly where it stopped looking.
A test with 2,000 users per arm reports effects 2.4 times too large. An A/A test checked ten times comes out significant 19% of the time. Twenty independent null metrics produce a winner 64% of the time and twelve null segments 46%. Four numbers, one cause, and the decisions that have to be made before the data arrives.
Entropy is not a summary of a distribution but a floor that the best code meets to the last decimal, the surcharge for using the wrong distribution is exactly the loss every classifier already minimises, and mutual information puts a hard ceiling on everything downstream of a sensor. Three results, each unusually sharp.
Estimators as random variables with distributions of their own, the case where the unbiased estimator is the worse one, what a confidence interval actually promises and the standard interval that delivers 87% where it advertises 95%, and what a p-value is a probability of - every figure computed exactly or by fixed-seed simulation.
Where the numbers in a Bayesian network or a Gaussian actually come from: the three-step maximum-likelihood recipe worked through on discrete and continuous parameters, the Beta prior that repairs what it does to an unseen event, naive Bayes and the single zero count that destroys it, and the EM algorithm for the case where the counts cannot be taken at all - with every figure computed rather than asserted.
The Bayes classifier nothing can beat and the error floor it leaves behind, k-nearest-neighbours as a nonparametric imitation with k as the flexibility dial, discriminant analysis and why a shared covariance forces a straight line, and the confusion matrix, thresholds and ROC curve that a single accuracy figure conceals - every number computed on simulated data where the optimum is known.
What changes when an agent gets noisy percepts instead of its state: the belief state that replaces it and the filtering update that maintains it, the exact reduction of a POMDP to an MDP over beliefs, the piecewise-linear convex value function that makes the reduction computable in principle, and the measured reasons it is not computable in practice - with the fixed point of belief, the alpha vectors and the value function all computed rather than asserted.
Why the widest slab between two classes is a good boundary, why insisting on a perfect one is self-defeating, how a budget for violations buys back stability, and how a kernel bends the boundary by working in a space it never has to build.
How to fit curved relationships without leaving least squares: basis functions, the constraints that turn a broken piecewise polynomial into a spline, the single extra column per knot that enforces them for free, and the roughness penalty that lets a curve choose its own flexibility.
Why a bet with positive expected monetary value can be rational to refuse, what the curvature of a utility function measures, and how to price an observation before buying it - including the common case where the honest price is zero.
What changes when you describe a problem as variables, domains and constraints instead of as a black box: commutativity that shrinks the tree for free, propagation that proves branches hopeless before searching them, and a measurement showing the most famous ordering heuristic does nothing on its own.
Why planning gets its own representation rather than being a footnote to search, how deleting parts of an action description produces a heuristic for free, and what a planning graph notices that per-goal heuristics systematically miss.
How two Markov assumptions turn an unbounded history into two small tables, the forward and backward recursions that answer every question about the present and the past, why the most likely sequence needs an algorithm of its own, and what changes when the state is a real number rather than a list.
How a graph and a few small tables stand in for a joint distribution with thousands of entries, how to answer a query against it exactly by enumeration and variable elimination, and what to do when exact inference is out of reach: rejection sampling, likelihood weighting, and Gibbs sampling, each worked through on the same two networks.
What changes when there is no response to predict: principal components as the direction of maximum variance, K-means and the local optima it settles into, hierarchical clustering and the linkage that decides the answer - and why none of the required choices can be validated the way a classifier can.
Turning a problem into a state space and letting an algorithm walk it: what completeness and optimality actually cost, why memory rather than time defeats breadth-first search, and the two conditions on a heuristic that make A* provably optimal.
Layers as parameterised transformations, the forward pass, and why depth and non-linearity are not optional: a proof that no single linear layer can compute XOR, and a two-layer network that does, worked entirely by hand.
How text becomes numbers a model can train on: building a vocabulary, why byte pair encoding never needs an unknown token, the embedding layer as a lookup that is provably one-hot times a matrix, and why position has to be added back in by hand.
Assembling a GPT from attention: multi-head projections, layer normalization worked by hand, why shortcut connections rescue the gradient, the 4x feed-forward expansion, and a parameter count that reproduces GPT-2 small at 124 million exactly.
Build probability from the ground up: possible worlds, the sample space, the two basic axioms, and the addition and multiplication rules, each derived rather than asserted, with worked numeric examples.
How next-word prediction turns unlabelled text into supervision, why cross entropy is just negative average log probability, what perplexity really measures, and why a model that completes text fluently still cannot follow an instruction.
Reasoning about what must be true: models and entailment worked by exhaustive enumeration, soundness and completeness, why propositional logic runs out of expressive power, and where first-order logic picks up.
Convolution defined properly, a Sobel edge detector worked by hand on a 5x5 image, why sliding one small kernel over an image beats a dense layer by five orders of magnitude in parameters, and what changed when kernels stopped being designed and started being learned.
Measuring uncertainty in bits: Shannon entropy and why the logarithm is base 2, information gain worked on a split, and how cross-entropy and KL divergence relate to entropy and to the loss functions used to train classifiers.
Derive Bayes' theorem from the definition of conditional probability, then work the base-rate example that fools almost everyone, twice: once with the formula and once by pure counting.
The setup behind every predictive model: estimating an unknown function f from data, the split between reducible and irreducible error, and why prediction and inference pull in different directions.
Learning to act well without a model of the world: temporal-difference updates, the Q-learning rule, exploration versus exploitation, and a run that recovers the planned optimum from experience alone.
The exact decomposition of expected test error into squared bias, variance, and irreducible noise, demonstrated numerically with a 2,000-run simulation where all three terms are measured separately and shown to add up.
How to plan when actions do not reliably do what you intend: states, transition models, rewards and discounting, the Bellman equation, and value iteration worked numerically to its fixed point.
Why training error is a biased estimate of test error, and how the validation set, leave-one-out, and k-fold approaches fix it, with a five-fold LOOCV computation worked out observation by observation.
Derive the least-squares coefficients by differentiating the residual sum of squares, then work a complete five-observation fit by hand: coefficients, fitted values, residuals, RSS, and R-squared, each verified numerically.
Why a straight line cannot model a probability, how the logistic function fixes it, and what the coefficients mean in log-odds, with a gradient-ascent step and a converged fit computed and checked numerically.
Adding a penalty on coefficient size to trade a little bias for a large reduction in variance, and why the L1 penalty sets coefficients exactly to zero while L2 only shrinks them, with both fitted numerically.
How recursive binary splitting builds a tree, why the Gini index beats accuracy as a splitting criterion, and how bagging and random forests turn a high-variance learner into a strong one, with the split arithmetic worked out.
How a neural network learns: the loss as a function of weights, gradient descent, and backpropagation as the chain rule applied backwards, with every partial derivative of a small network computed by hand and checked against autograd.
Queries, keys, and values built from the ground up: why attention exists, how scaled dot-product attention is computed, why it is divided by the square root of the dimension, and how causal masking works, with every matrix computed and checked.
How a program plays a game against an opponent who is trying to beat it: the minimax value, why alpha-beta pruning reaches the same answer while examining fewer nodes, and a game tree pruned move by move.
Strategic reasoning when players are not strictly opposed: dominant strategies, the prisoner's dilemma worked from its payoff matrix, Nash equilibrium, Pareto optimality, and why equilibrium and efficiency can conflict.