Gradient Boosting

Gradient boosting builds a prediction by adding many small trees one after another. Each new tree is trained on the mistakes the current model is still making.

The idea

Picture a team of painters restoring a mural. The first one blocks in the rough colours. Each painter after that only touches up the spots that still look wrong, and only lightly. After enough passes the mural is sharp, even though no single painter did much.

Each tree here is deliberately weak, just a few splits deep. The learning rate shrinks every correction, so the model creeps towards the data rather than lunging at it.

How it learns

  1. Start by predicting the mean of all y values everywhere.
  2. Compute residuals: how far each point sits above or below the current prediction.
  3. Fit a small regression tree on x to those residuals.
  4. Add the tree's output, scaled by the learning rate, to the prediction.
  5. Repeat until the chosen number of trees has been added.

Reading the picture

Press play and watch the curve begin as a flat line at the mean, then gain detail with every tree. Early trees capture the big shape; later ones refine small bumps.

Turn on Show residuals to see exactly what the next tree is about to be trained on. As training goes on these segments shrink, and the Training MSE in the stats falls with them.

Try this

  1. On Wave, step through training one tree at a time. Notice the first few trees do most of the work.
  2. Set Learning rate to 1 and then to 0.05 with the same Number of trees. A high rate fits fast but jumpy; a low rate is smoother but may not finish the job.
  3. Raise Tree depth to 4 and Number of trees to 150 on a noisy Wave. Watch the curve start chasing individual points: boosting can overfit too.
  4. Load Outliers and keep Show residuals on. The outliers keep their long residuals and keep tugging at later trees.

The maths

Start

F0(x)=yˉF_0(x) = \bar{y}

Residuals

ri=yi−Fm−1(xi)r_i = y_i - F_{m-1}(x_i)

Fit a tree

hm=regression tree fitted to (xi,ri)h_m = \text{regression tree fitted to } (x_i, r_i)

Update

Fm(x)=Fm−1(x)+η⋅hm(x)F_m(x) = F_{m-1}(x) + \eta \cdot h_m(x)

Final model

F(x)=yˉ+η∑mhm(x)F(x) = \bar{y} + \eta \sum_m h_m(x)

Strengths and weaknesses

Good at

  • Very accurate on structured, tabular data.
  • Builds complex shapes from simple pieces.
  • Learning rate and tree count give fine control.

Struggles with

  • Trees are added in sequence, so training is slow to parallelise.
  • Several interacting knobs to tune.
  • Will happily fit noise if run too long.