Law of Large Numbers

The idea

Roll a die once and you might get a 6. Average ten rolls and you will probably be somewhere between 2.5 and 4.5. Average ten thousand and you will be within a whisker of 3.5. Individual outcomes stay random, but their average becomes predictable.

Each line on the chart is one person rolling and keeping a running average. Early on the lines swing wildly. As $n$ grows they all squeeze together onto the true mean.

How fast?

The spread of the average shrinks like $\sigma/\sqrt{n}$. To halve the error you need four times as many draws; to get one more decimal place you need a hundred times as many. The shaded band shows $\mu \pm 2\sigma/\sqrt{n}$, where about 95% of the averages should sit at each $n$.

Roll it yourself

Here is a single die rolled one throw at a time. The face counts start out lumpy, and the running average jumps around before settling on the expected value. Load the die and it settles somewhere else, on the new expected value, which shows the law has nothing to do with fairness, only with independence and a finite mean.

When it fails

The law needs the expected value to exist. The Cauchy distribution (Student's $t$ with $\nu = 1$) has tails so heavy that its mean is undefined. Its running averages never settle: every so often a single huge draw yanks the average somewhere new.

Try this

1. Run the Fair die with 10 paths. Notice how wild the first 20 rolls are and how tight things look by 1,000.
2. Switch to Exponential. The paths overshoot upwards more than downwards early on because the distribution is skewed, but they still converge.
3. Pick Cauchy and run it several times. The paths jump and never settle, because there is no mean for them to converge to.

The maths