A feed-forward network in NumPy with hand-derived backpropagation, validated against numerical gradients so the calculus is proven rather than trusted.
A multilayer perceptron written with nothing but NumPy, built to make backpropagation concrete. The forward pass, the loss, and the analytic gradients for every layer are derived by hand and then implemented directly, and the central discipline of the project is gradient checking: each analytic gradient is compared against a finite-difference estimate, and the implementation is only accepted when the relative error falls to the order of 1e-7. That check is what turns backpropagation from a formula the reader has seen into one they have verified. The network is then trained on XOR - chosen because no single linear layer can represent it, which makes the need for a hidden layer a demonstrated fact rather than an assertion. Implementation is in progress and no source repository has been published yet.
Layers as parameterised transformations, the forward pass, and why depth and non-linearity are not optional: a proof that no single linear layer can compute XOR, and a two-layer network that does, worked entirely by hand.
How a neural network learns: the loss as a function of weights, gradient descent, and backpropagation as the chain rule applied backwards, with every partial derivative of a small network computed by hand and checked against autograd.