Bias–variance trade-off
Out-of-sample error splits into three parts: bias from a model that is too simple, variance from a model that chases noise, and noise nobody can remove. Making a model more complex trades one for the other.
01The formula
- a new outcome, not used in fitting
- the true relationship you are trying to learn
- the model's forecast, which depends on the training sample
- how far the average forecast is from the truth: error from a model too simple
- how much the forecast jumps between training samples: error from a model too flexible
- noise no model can predict
02See it move
03Where it is used
- Quant researcherChooses how many signals or how deep a model to use by finding where test error stops falling.
- Portfolio managerPrefers fewer, robust signals when the history is short, because variance dominates on small samples.
- Risk managerQuestions models whose backtest fit is much better than their live performance: the gap is variance.
04Why traders use it
Market data are short and noisy, so variance is usually the bigger danger. This decomposition is why simpler models and shrinkage so often beat clever ones in trading.
05Worked example
Thirty signals have true effects fading like j^(−1.00); with noise 1.00 and 100 training points, fitting d of them costs bias² for the ones left out and d·σ²/n of variance. The best choice is d = 9, with bias² 0.0724 and variance 0.0900; expected test error, noise included, is 1.162.
06How it is built
- Write .
- Square and take expectations. The cross terms vanish because the noise in and the sampling noise in are independent and have mean zero.
- What is left is .
- In this model, fitting the first of 30 independent signals by least squares gives bias² and variance .
07The common mistake
Judging a model by its training error. Training error only ever falls as you add complexity; the variance part shows up only on data the model has not seen.
08In Python and LaTeX
import numpy as np
sigma, n, alpha = 1.0, 100, 1.0
j = np.arange(1, 31)
bias2 = np.array([np.sum(j[d:] ** (-2 * alpha)) for d in range(1, 31)])
var = np.arange(1, 31) * sigma**2 / n
print('best d:', np.argmin(bias2 + var) + 1) # 9E\big[(y - \hat f(x))^2\big] = \underbrace{\big(f(x) - E[\hat f(x)]\big)^2}_{\text{bias}^2} + \underbrace{\operatorname{Var}\big(\hat f(x)\big)}_{\text{variance}} + \sigma^209Where it fits
Rather not code the maths yourself? Describe your trading idea and let AI build the bot.
Build a bot with AgenKitThe worked example and the model run the same code, so every number matches. Models simplify markets on purpose. This is education, not financial advice.