Quadratic Discriminant Analysis

QDA fits a separate bell-shaped Gaussian to each class, each with its own size, stretch and tilt. Comparing differently shaped clouds produces curved boundaries.

The idea

LDA uses one cookie cutter for every class. QDA gives each class its own: a tight circle for one, a long tilted oval for another. A point belongs to whichever class finds it least surprising.

When a narrow class sits next to a wide one, the wide one wins far out in every direction, and the narrow one only wins close to home. The boundary between them curls into an ellipse, a parabola or a hyperbola.

How it learns

  1. Compute each class's mean.
  2. Compute each class's own covariance from its points.
  3. Add a small amount to the diagonal (covariance regularisation) so no ellipse collapses.
  4. Set each class prior to its share of the points.
  5. Classify with Bayes' rule using each class's own Gaussian.

Reading the picture

With Show class Gaussians on, every class draws its own ellipses, so you can see one class is round while another is stretched. The background shows the most likely class, shaded by confidence.

Boundaries are quadratic curves. Watch for regions that wrap around a tight class, or for a wide class reappearing far away on the other side of a narrow one.

Try this

  1. On Circles, see how the inner class gets a small round Gaussian and the outer a big one. The boundary becomes a ring.
  2. Switch to Blobs and drag points of one class into a long diagonal streak. Its ellipse rotates to match and the boundary bends.
  3. Remove points from a class until only three remain, then raise Covariance regularisation. The fragile, thin ellipse inflates to something sensible.
  4. Try Spiral. A single Gaussian per class cannot describe a spiral, so QDA's limits show.

The maths

Class model

p(x∣k)=N(x; μk,Σk)p(x \mid k) = \mathcal{N}(x;\ \mu_k, \Sigma_k)

Covariance

Σk=1nk−1∑i∈k(xi−μk)(xi−μk)⊤+εI\Sigma_k = \frac{1}{n_k - 1}\sum_{i \in k} (x_i - \mu_k)(x_i - \mu_k)^\top + \varepsilon I

Discriminant

δk(x)=−12log⁡∣Σk∣−12(x−μk)⊤Σk−1(x−μk)+log⁡πk\begin{aligned} \delta_k(x) = &-\tfrac{1}{2}\log\lvert\Sigma_k\rvert - \tfrac{1}{2}(x - \mu_k)^\top \Sigma_k^{-1} (x - \mu_k) \\ &+ \log \pi_k \end{aligned}

Posterior

P(k∣x)=softmax⁡(δ1(x),…,δK(x))kP(k \mid x) = \operatorname{softmax}\big(\delta_1(x), \dots, \delta_K(x)\big)_k

Strengths and weaknesses

Good at

  • Curved boundaries with no feature engineering.
  • Adapts to classes with very different spreads.
  • Closed-form and fast to fit.

Struggles with

  • Needs more data than LDA: three covariance numbers per class.
  • Unstable when a class has very few points.
  • Still assumes each class is one Gaussian blob.