Linear Regression from First Principles

Linear regression is old, simple, and still the right first thing to try. It is also the cleanest place to see the pattern every other supervised method follows: write down a measure of how badly the model fits, then choose the parameters that minimise it. Here that minimisation can be done exactly, in closed form, with calculus you already have.

A. The model

With a single predictor, we assume

$$Y \approx \beta_0 + \beta_1 X ,$$

where $\beta_0$ is the intercept and $\beta_1$ the slope. Together they are the model's coefficients. Estimating them from data gives $\hat\beta_0$ and $\hat\beta_1$, and predictions

$$\hat y_i = \hat\beta_0 + \hat\beta_1 x_i .$$

The $i$-th residual is $e_i = y_i - \hat y_i$, the gap between what we saw and what we predicted.

B. What "best fit" means

We need a single number for how badly a candidate line fits. The standard choice is the residual sum of squares:

$$\mathrm{RSS} = \sum_{i=1}^{n} e_i^2 = \sum_{i=1}^{n}\big(y_i - \beta_0 - \beta_1 x_i\big)^2 .$$

Squaring does two things: it makes positive and negative misses count the same, and it penalises one large miss more heavily than several small ones. It is also differentiable everywhere, which is what makes the closed-form solution below possible.

Least squares chooses $\hat\beta_0, \hat\beta_1$ to minimise RSS.

C. Deriving the coefficients

RSS is a smooth function of two variables, so its minimum is where both partial derivatives vanish.

With respect to the intercept:

$$\frac{\partial\,\mathrm{RSS}}{\partial \beta_0} = \sum_{i=1}^n 2\big(y_i - \beta_0 - \beta_1 x_i\big)(-1) = 0 \;\Longrightarrow\; \sum_{i=1}^n \big(y_i - \beta_0 - \beta_1 x_i\big) = 0 .$$

Dividing by $n$ gives $\bar y - \beta_0 - \beta_1 \bar x = 0$, so

$$\hat\beta_0 = \bar y - \hat\beta_1 \bar x .$$

Two things follow immediately: the residuals sum to zero, and the fitted line always passes through $(\bar x, \bar y)$.

With respect to the slope:

$$\frac{\partial\,\mathrm{RSS}}{\partial \beta_1} = \sum_{i=1}^n 2\big(y_i - \beta_0 - \beta_1 x_i\big)(-x_i) = 0 .$$

Substituting $\beta_0 = \bar y - \beta_1\bar x$ and rearranging yields

$$\hat\beta_1 = \frac{\sum_{i=1}^n (x_i - \bar x)(y_i - \bar y)}{\sum_{i=1}^n (x_i - \bar x)^2} .$$

The numerator is the sample covariance of $x$ and $y$ (up to a constant) and the denominator the sample variance of $x$. So the slope is how much $x$ and $y$ move together, scaled by how much $x$ moves on its own - which is exactly what a slope ought to be.

D. A complete worked fit

Five observations:

Step 1 - the means.

$$\bar x = \frac{1+2+3+4+5}{5} = 3, \qquad \bar y = \frac{2+4+5+4+5}{5} = 4 .$$

Step 2 - the deviation products.

Step 3 - the coefficients.

$$\hat\beta_1 = \frac{6}{10} = 0.6 = \tfrac{3}{5}, \qquad \hat\beta_0 = 4 - 0.6 \times 3 = 4 - 1.8 = 2.2 = \tfrac{11}{5} .$$

The fitted line is

$$\hat y = 2.2 + 0.6\,x .$$

Step 4 - fitted values and residuals.

The residuals sum to exactly zero, as the derivation promised, and the line passes through $(3, 4) = (\bar x, \bar y)$ - check row 3.

$$\mathrm{RSS} = 2.40 .$$

E. How good is the fit?

RSS alone is uninterpretable: it depends on the units and the sample size. Compare it instead against the total sum of squares, the error of the best possible constant prediction $\bar y$:

$$\mathrm{TSS} = \sum (y_i - \bar y)^2 = 4 + 0 + 1 + 0 + 1 = 6 .$$

Then

$$R^2 = 1 - \frac{\mathrm{RSS}}{\mathrm{TSS}} = 1 - \frac{2.4}{6} = 0.6 .$$

The predictor explains 60% of the variability in $y$; the remaining 40% it does not. $R^2$ lies in $[0,1]$ for a model fitted with an intercept, and is unitless, which is what makes it comparable across problems.

$R^2$ never decreases when you add a predictor, even a column of pure noise, because the extra freedom can only reduce RSS. It therefore cannot be used to choose between models of different sizes - that is what cross-validation is for.

F. Verifying every number

Running it prints the table exactly - slope $0.6$, intercept $2.2$, residuals summing to $0$ (to machine precision), $\mathrm{RSS} = 2.4$, $\mathrm{TSS} = 6.0$, $R^2 = 0.6$ - and sklearn: 2.2 0.6 0.6, an independent confirmation.

G. What the model assumes

Least squares always returns an answer. Whether it is meaningful depends on assumptions worth stating plainly:

• Linearity. The relationship really is a straight line. If it curves, the fit is biased no matter how much data you collect - the failure mode from The Bias-Variance Tradeoff.
• Independent errors. Correlated residuals (time series, clustered data) leave coefficients usable but make standard errors far too small.
• Constant error variance. If spread grows with $x$, the fit over-weights the noisy region.
• Influential points. With $n = 5$, a single outlier moves the line substantially - as LOOCV fold 1 showed, dropping $(1,2)$ moved the prediction there from $2.8$ to $4.0$, a shift of $1.2$.

Plotting residuals against fitted values checks the middle two faster than any formal test.

Key takeaways

• Least squares minimises RSS, and setting both partial derivatives to zero gives closed-form coefficients - no iteration required.
• $\hat\beta_1$ is the covariance of $x$ and $y$ over the variance of $x$; $\hat\beta_0$ forces the line through $(\bar x, \bar y)$.
• The derivation guarantees the residuals sum to zero.
• Our worked fit: $\hat y = 2.2 + 0.6x$, $\mathrm{RSS} = 2.4$, $\mathrm{TSS} = 6$, $R^2 = 0.6$.
• $R^2$ never decreases when predictors are added, so it cannot compare models of different sizes.

What's next

Squared error is the wrong target when the response is a category rather than a number - a fitted line will happily predict a probability of $1.4$. Adapting the linear model to classification is Logistic Regression and Classification.