Perceptron

The perceptron is the simplest learning machine: a straight line that separates two classes. Every time it misclassifies a point, it nudges the line towards getting that point right.

The idea

Imagine a referee drawing a chalk line between two teams. Whenever a player is standing on the wrong side, the referee shifts the line a little towards fixing that one mistake. Eventually, if a clean line exists, nobody is on the wrong side.

It only ever reacts to mistakes. Points already on the correct side are ignored, no matter how close to the line they are. That makes it fast, but it also means the final line is just the first one that works, not the best one.

How it learns

  1. Start with a blank line (all weights zero).
  2. Visit the points in a shuffled order.
  3. If a point is on the wrong side, add it (scaled by the learning rate) to the weights, pointing the line towards its class.
  4. After a full pass, count the mistakes.
  5. Stop when a pass has zero mistakes, or give up after 300 passes.

Reading the picture

The line is the current boundary, with class A on one side and class B on the other. The background shading fades near the line and strengthens with distance from it. It is a visual cue, not a true probability: the perceptron only knows which side you are on.

Each animation step is one pass through the data. Mistakes last pass in the stats counts the corrections made; when it hits zero the data has been separated and training stops.

Try this

  1. On Separable, press step repeatedly and watch the line jerk into place after a handful of passes.
  2. Load XOR and press play. No straight line can separate it, so the line keeps flipping until it hits the pass limit.
  3. On Separable, change Learning rate and retrain. Starting from zero, the rate only rescales the weights, so the line follows the same path.
  4. Add a single class A point deep inside class B's side. One mislabelled point is enough to stop convergence.

The maths

Prediction

y^=sign⁡(w⊤x+b),y∈{−1,+1}\hat{y} = \operatorname{sign}(w^\top x + b), \quad y \in \{-1, +1\}

Mistake

yi(w⊤xi+b)≤0y_i (w^\top x_i + b) \le 0

Update

w←w+η yixi,b←b+η yiw \leftarrow w + \eta\, y_i x_i, \quad b \leftarrow b + \eta\, y_i

Distance to line

d(x)=w⊤x+b∥w∥d(x) = \frac{w^\top x + b}{\lVert w\rVert}

Strengths and weaknesses

Good at

  • Tiny, fast and easy to understand.
  • Guaranteed to converge if the classes are linearly separable.
  • The ancestor of every modern neural network.

Struggles with

  • Only straight boundaries, only two classes.
  • Never settles if the data is not separable.
  • Stops at any working line, not the widest-margin one.