Linear & Polynomial Regression

Linear regression draws the line (or curve) that sits as close as possible to your points, measured by squared vertical distance. Raise the polynomial degree and the same method can fit bends and wiggles.

The idea

Imagine every point is tied to the line with a little vertical spring. Long springs pull much harder than short ones, because the pull grows with the square of the stretch. The line settles where the total tension is as small as possible.

A polynomial fit is the same trick with extra ingredients. Instead of only xx, the model may use x2x^2, x3x^3 and so on, so the curve can bend. The maths does not change: it is still a weighted sum, just of more features.

How it learns

  1. Turn each xx into a row of features: 1,x,x21, x, x^2, up to the chosen degree.
  2. Measure the error as the mean of squared residuals, the vertical gaps between points and curve.
  3. Optionally add a ridge penalty that charges for large weights (the intercept is never penalised).
  4. Solve the resulting linear equations directly. There is no training loop: the best weights come out in one step.

Reading the picture

The coloured curve is the fitted model. With Show residuals on, each point gets a thin segment to the curve: those are the errors the model is trying to shrink, and the MSE in the stats is their average squared length.

R2R^2 tells you how much better the fit is than a flat line at the mean yy. For degree 3 or below the stats also print the fitted equation, so you can read the slope and intercept directly.

Try this

  1. On Line, leave Polynomial degree at 1 and drag a single point far up. Watch the whole line tilt towards it: squared error makes one outlier very loud.
  2. Switch to Curve and step Polynomial degree from 1 to 2. R2R^2 should jump, because the data really is a parabola.
  3. Pick Wave, lower the point count, and push Polynomial degree to 12. The curve threads every point and swings wildly between them. Now raise Ridge penalty (λ\lambda) and watch it calm down.
  4. Try Outliers with degree 1 and compare the fit with and without the stray points (shift-click to erase them).

The maths

Model

y^=w0+w1x+w2x2+⋯+wdxd\hat{y} = w_0 + w_1 x + w_2 x^2 + \dots + w_d x^d

Loss

L(w)=1n∑i(yi−y^i)2+λ∑j=1dwj2L(w) = \frac{1}{n}\sum_i (y_i - \hat{y}_i)^2 + \lambda \sum_{j=1}^{d} w_j^2

Closed form

w=(X⊤X+nλI′)−1X⊤y(I′ skips the intercept)w = (X^\top X + n\lambda I')^{-1} X^\top y \quad (I' \text{ skips the intercept})

Residual

ri=yi−y^ir_i = y_i - \hat{y}_i

R-squared

R2=1−∑iri2∑i(yi−yˉ)2R^2 = 1 - \frac{\sum_i r_i^2}{\sum_i (y_i - \bar{y})^2}

Strengths and weaknesses

Good at

  • Fast, exact and easy to interpret.
  • Coefficients tell you how yy changes with xx.
  • Polynomial features add flexibility without new maths.

Struggles with

  • Squared error lets outliers drag the fit around.
  • High degrees overfit and explode at the edges.
  • You must choose the shape (degree) up front.