Logistic Regression

Logistic regression gives each class a linear score and turns those scores into probabilities. It learns the scores by gradient descent, making the true class more likely at every step.

The idea

Think of each class as a candidate in an election, with a score that rises as you move in its favoured direction across the plane. Softmax converts the scores into vote shares that add up to one. Where two candidates tie, you get the boundary.

On its own the score is linear, so boundaries are straight lines. Feed it extra features like x2x^2, xyxy and y2y^2, and the same linear model can draw circles, ellipses and curves in the original plane.

How it learns

  1. Start with all weights at zero, so every class has equal probability everywhere.
  2. Compute each point's predicted probabilities with softmax.
  3. Measure the log loss: how surprised the model is by the true labels.
  4. Compute the gradient over all points at once and step the weights downhill by the learning rate.
  5. Repeat until the gradient is tiny or the epoch limit is reached.

Reading the picture

The background colour shows the most likely class, and its strength shows confidence. Pale bands mark places where the model is unsure. The boundary line is where two classes are equally likely.

Press play to watch the boundary rotate and slide into place. Log loss in the stats should fall steadily; if it bounces around, the learning rate is too high.

Try this

  1. On Blobs, press play and watch the boundaries settle. Then add a class C point deep in A's territory and see how far it can pull the line.
  2. Switch to Circles with Feature degree 1. No straight line works. Set Feature degree to 2 and the boundary becomes a ring.
  3. Try XOR at Feature degree 2. The xyxy feature alone is enough to solve it.
  4. On Moons, set Feature degree to 6 and Regularisation (λ\lambda) to the far left, then slide λ\lambda up. Watch an overly twisty boundary smooth out.

The maths

Features

ϕ(x)=[1,x1,x2,x12,x1x2,x22,… ]\phi(x) = [1, x_1, x_2, x_1^2, x_1 x_2, x_2^2, \dots]

Scores

zk=wk⊤ϕ(x)z_k = w_k^\top \phi(x)

Softmax

P(y=k∣x)=ezk∑jezjP(y = k \mid x) = \frac{e^{z_k}}{\sum_j e^{z_j}}

Loss

L=−1n∑ilog⁡P(yi∣xi)+λ2∑k∥wk∥2(bias not penalised)L = -\frac{1}{n}\sum_i \log P(y_i \mid x_i) + \frac{\lambda}{2}\sum_k \lVert w_k\rVert^2 \quad (\text{bias not penalised})

Gradient step

wk←wk−η[1n∑i(pik−1[yi=k]) ϕ(xi)+λwk]w_k \leftarrow w_k - \eta \left[ \frac{1}{n}\sum_i \big(p_{ik} - \mathbf{1}[y_i = k]\big)\,\phi(x_i) + \lambda w_k \right]

Strengths and weaknesses

Good at

  • Outputs calibrated-looking probabilities, not just labels.
  • Weights are interpretable and training is stable.
  • Feature expansion makes it surprisingly flexible.

Struggles with

  • Linear in its features: curves need hand-made features.
  • High feature degrees overfit without regularisation.
  • Gradient descent can be slow with a poor learning rate.