Linear Discriminant Analysis

LDA models each class as a bell-shaped Gaussian cloud, with every class sharing the same shape and tilt. A new point goes to the class whose cloud explains it best, which gives straight boundaries.

The idea

Imagine stamping the same oval cookie cutter onto the plane once per class, centred on that class's average point. To classify a new point, ask which stamp it sits most comfortably inside, allowing for how common each class is.

Because every class uses the identical oval, the curved parts cancel when you compare two classes. What is left is a straight-line boundary, even though the model is built from Gaussians.

How it learns

  1. Compute the mean of each class.
  2. Measure how points spread around their own class mean, and pool that spread into one shared covariance.
  3. Optionally shrink the shared covariance towards a circle.
  4. Set each class prior to its share of the points.
  5. Classify with Bayes' rule: pick the class with the highest prior ×\times likelihood.

Reading the picture

With Show class Gaussians on, each class gets ellipses around its mean. They all have the same shape and orientation, only the centres differ. The background shows the most probable class, shaded by confidence.

The boundaries are always straight lines. They are not simply halfway between the means: the shared tilt of the ellipses rotates them, and class priors shift them towards the rarer class.

Try this

  1. On Blobs, drag the points of one class into a long streak. Every class's ellipse changes shape together, because they all share one covariance.
  2. Load Circles. LDA can only draw straight lines, so it fails. Compare with QDA on the same data.
  3. Erase most points from one class until only three or four remain, then raise Shrinkage. The ellipses become rounder and the boundary steadier.
  4. Add many extra points to class A on Blobs. The boundary slides towards B as A's prior grows.

The maths

Class model

p(x∣k)=N(x;μk,Σ)p(x \mid k) = \mathcal{N}(x; \mu_k, \Sigma)

Shared covariance

Σ=1n−K∑k∑i∈k(xi−μk)(xi−μk)⊤\Sigma = \frac{1}{n - K} \sum_k \sum_{i \in k} (x_i - \mu_k)(x_i - \mu_k)^\top

Shrinkage

Σα=(1−α) Σ+α⋅tr⁡Σ2⋅I\Sigma_\alpha = (1 - \alpha)\, \Sigma + \alpha \cdot \frac{\operatorname{tr} \Sigma}{2} \cdot I

Discriminant

δk(x)=x⊤Σ−1μk−12μk⊤Σ−1μk+log⁡πk\delta_k(x) = x^\top \Sigma^{-1} \mu_k - \frac{1}{2} \mu_k^\top \Sigma^{-1} \mu_k + \log \pi_k

Posterior

P(k∣x)=softmax⁡k(δk(x))P(k \mid x) = \operatorname{softmax}_k\big(\delta_k(x)\big)

Strengths and weaknesses

Good at

  • Few parameters, so it works with little data.
  • Fast, closed-form fit with no tuning loop.
  • Gives probabilities and interpretable class shapes.

Struggles with

  • Boundaries are always straight.
  • Assumes every class has the same spread.
  • Sensitive to outliers that distort the means.