What a Network Computes

A neural network is a composition of simple functions with adjustable constants. Nothing about that sentence is metaphorical, and this lesson makes it concrete before any training is involved.

A layer is a function with parameters

The standard fully connected - dense - layer computes

$$\mathbf{h} = \sigma(W\mathbf{x} + \mathbf{b}),$$

where $W$ is a weight matrix, $\mathbf{b}$ a bias vector, and $\sigma$ an elementwise nonlinearity. Two distinct things happen: an affine map $W\mathbf{x} + \mathbf{b}$, then a nonlinearity applied to each coordinate independently.

Chollet's framing is useful: layers are the building bricks of deep learning, and the shape of the data selects the brick. Vectors go to dense layers; sequences in 3D tensors go to recurrent or 1D convolution layers; images in 4D tensors go to 2D convolution layers.

Counting parameters

A dense layer from $n$ inputs to $m$ outputs holds an $m \times n$ matrix plus a bias per output unit:

$$\text{parameters} = mn + m .$$

From 4 inputs to 3 outputs that is $3 \times 4 + 3 = 15$. Forgetting the bias vector is the usual slip and gives 12.

Why the nonlinearity is not optional

Suppose we drop $\sigma$ and stack three dense layers:

$$W_3\big(W_2(W_1\mathbf{x} + \mathbf{b}_1) + \mathbf{b}_2\big) + \mathbf{b}_3 .$$

Multiply it out. The weight matrices collapse into a single product $W = W_3W_2W_1$ and the biases collapse into a single vector, leaving $W\mathbf{x} + \mathbf{b}$ - an affine map. Three layers, and the family of functions is exactly what one layer could already express.

Depth alone buys nothing. A composition of affine maps is affine. Whatever expressive power depth provides arrives only once a nonlinearity sits between the layers, which is why $\sigma$ is structural rather than decorative.

ReLU, precisely

The standard choice is the rectified linear unit:

$$\text{ReLU}(z) = \max(0, z).$$

Raschka describes it plainly: it thresholds negative inputs to 0. That is the whole definition. It does not squash values into $(0,1)$ - that is the logistic sigmoid - and it does not normalise anything.

Its effect on the backward pass matters as much as on the forward one: the derivative is $1$ where $z > 0$ and $0$ where $z < 0$, so a unit that is inactive for an example passes no gradient back through it for that example.

A worked forward pass

Two inputs, two hidden units, one output:

$$\mathbf{x} = \begin{bmatrix}1\\2\end{bmatrix}, \quad W^{(1)} = \begin{bmatrix}0.1&0.3\\0.2&0.4\end{bmatrix}, \quad W^{(2)} = \begin{bmatrix}0.5&-0.5\end{bmatrix}$$ $$\mathbf{z}^{(1)} = \begin{bmatrix}0.1(1)+0.3(2)\\0.2(1)+0.4(2)\end{bmatrix} = \begin{bmatrix}0.7\\1.0\end{bmatrix}$$

Both entries are positive, so ReLU leaves them unchanged, and

$$\hat y = 0.5(0.7) - 0.5(1.0) = -0.15 .$$

That single number is what the network currently believes. The next lesson asks how wrong it is and pushes that error backwards to every weight.

The figure below is that same network, sliced: the output against the first input, with the second on a slider. Take the nonlinearity away and the slice is a straight line, because the two layers multiply out to the single row [-0.05, -0.05]. The distance from affine drops to zero at machine precision, and the collapse is something you can check rather than take on faith.

Put ReLU back and notice where you are standing. At the worked input (1, 2) both units are on, so the rectifier does nothing at all, and the network and its collapsed form both return -0.15. The nonlinearity earns its keep only across a boundary: at (2, -1.5) both units are off and the answers are 0 and -0.025.

So depth does not buy a curve. It buys a set of regions, each of which is still an affine map, and the kinks in the slice are exactly where a pre-activation crosses zero. That is worth carrying into the next lesson, because it is also why a unit that is off passes no gradient back.

Continue with backpropagation and gradient descent for the general treatment.