The law of large numbers and the central limit theorem

Probability runs forwards: given the coin, what will the flips look like? Statistics runs backwards: given the flips, what is the coin? You never see the true parameters of a market, only data. Two theorems connect the two directions, and almost every number a quant reports leans on them.

The statistician’s set-up

Treat each observation as a random variable, and assume a family for it (Bernoulli, Poisson, normal…). Usually also assume the observations are i.i.d.: independent and identically distributed.
Link the unknown parameter to the data with an estimator. If each $X_i$ is Bernoulli $(p)$ then $E[X_i] = p$ , so estimate $p$ with the sample average: $\hat p = \bar X_n$ .
Say how confident you are. If $\hat p = 0.55$ , is the truth in $[0.5, 0.6]$ ? $[0.54, 0.56]$ ? The next lesson answers that.

Law of large numbers

\bar X_n = \frac{1}{n}\sum_{i=1}^n X_i \;\xrightarrow[n\to\infty]{}\; \mu \quad\text{(in probability, and almost surely).}

Averages converge to the true mean. That is what makes the estimator in step 2 sensible: for every common distribution the mean is a simple function of the parameter, so a long enough average pins the parameter down.

Figure 6.1Running averages settle down

Eight people each flip a coin 1,000 times and track their running share of heads. Early on they disagree wildly; by the end they all hug p. The green funnel shrinks like 1/√n.

Central limit theorem

The LLN says the average gets close. The CLT says how close, and what the error looks like:

\sqrt{n}\,\frac{\bar X_n - \mu}{\sigma} \;\xrightarrow[n\to\infty]{(d)}\; \mathcal N(0, 1).

Whatever the distribution of a single draw (as long as its variance is finite), the error of the average is approximately normal with standard deviation $\sigma/\sqrt n$ . This is why the bell curve is everywhere: many measured quantities are sums of lots of small independent effects.

Figure 6.2Galton board, in 3D

Each ball bounces left or right at every row, so its final bin is a sum of independent coin flips: a binomial. Pile up enough balls and the binomial looks like the normal curve. Make the bounce lopsided and it still does, just shifted.

Figure 6.3CLT sampler

Pick an ugly source distribution (shown top right) and the number of draws per average. The histogram is the averages, shifted by μ and scaled by σ/√n. At n = 1 it is the source itself; by n = 30 almost everything sits on the normal curve. Very skewed sources, like the lopsided coin, need a larger n.

In code

import numpy as np
rng = np.random.default_rng(4)

# Averages of n exponential draws, standardised: approaches N(0, 1)
n, reps = 30, 200_000
x = rng.exponential(1.0, size=(reps, n))
z = (x.mean(axis=1) - 1.0) / (1.0 / np.sqrt(n))
print(z.mean(), z.std())                          # ≈ 0, ≈ 1
print(np.mean(np.abs(z) < 1.96))                  # ≈ 0.95

# How long until a Sharpe-1 strategy shows t = 2?
sharpe = 1.0
years = (2 / sharpe) ** 2
print(years)                                      # 4.0

Check yourself

You estimate a mean from 100 i.i.d. observations and the standard error is 0.4. Roughly how many observations do you need for a standard error of 0.1?