Regression Tree

A regression tree chops the x-axis into intervals and predicts the average y inside each one. The result is a staircase that can follow almost any shape.

The idea

Think of playing twenty questions with a number line. "Is xx below 0.30.3?" Yes. "Is it below −0.2-0.2?" No. After a few questions you have narrowed xx down to a slice, and you predict the typical yy in that slice.

Each question is chosen greedily: the tree picks whichever cut makes the two halves as internally consistent as possible, then asks again inside each half.

How it learns

  1. Start with all points in one group, predicting their mean yy.
  2. Try every cut point on xx and score it by the total squared error of the two halves.
  3. Keep the best cut and repeat inside each half.
  4. Stop when Max depth is reached or a split would leave fewer than Min points per leaf on a side.
  5. Each final group (a leaf) predicts the mean yy of its points.

Reading the picture

The curve is a set of flat steps, one per leaf. Every jump marks a split threshold. Count the steps and you get the Leaves figure in the stats.

Turn on Show residuals to see how far each point sits from its step. Deeper trees make more, narrower steps, and the Training MSE drops towards zero as the tree starts memorising noise.

Try this

  1. On Wave, set Max depth to 1. You get a single split: two flat steps. Add one level at a time and watch the steps double.
  2. Push Max depth to 10 with Min points per leaf at 1. The steps chase every noisy point. Now raise Min points per leaf to 8 and see the tree pull itself back together.
  3. Try Line at depth 3. A straight line becomes a staircase: trees cannot extrapolate a slope.
  4. Load Outliers. A stray point only affects its own leaf rather than tilting the whole curve.

The maths

Leaf prediction

y^=1∣R∣∑i∈Ryi\hat{y} = \frac{1}{\lvert R\rvert}\sum_{i \in R} y_i

Split cost

SSE⁡(L)+SSE⁡(R),SSE⁡(S)=∑i∈S(yi−yˉS)2\operatorname{SSE}(L) + \operatorname{SSE}(R), \quad \operatorname{SSE}(S) = \sum_{i \in S} (y_i - \bar{y}_S)^2

Threshold

t=midpoint between neighbouring sorted xi valuest = \text{midpoint between neighbouring sorted } x_i \text{ values}

Max leaves

leaves≤2depth\text{leaves} \le 2^{\text{depth}}

Strengths and weaknesses

Good at

  • Handles curves and jumps with no feature engineering.
  • Robust to outliers: they only affect one leaf.
  • Easy to explain as a list of rules.

Struggles with

  • Predictions are blocky, never smooth.
  • Cannot extrapolate beyond the training range.
  • Deep trees overfit and change a lot with small data changes.