Linear regression: β, the hat matrix, t and F

If you learn one formula by heart for quant interviews, make it this one. Linear regression is the workhorse of quant research: factor models, hedge ratios, signal testing and most first-pass alpha models are regressions.

The closed form

Stack the $N$ observations into a matrix $X$ ($N \times (p+1)$, with a column of ones for the intercept) and a vector $y$. Minimising $\operatorname{RSS}(\beta) = (y - X\beta)^\top(y - X\beta)$ means setting its gradient to zero:

The fitted values are $\hat y = X\hat\beta = X(X^\top X)^{-1}X^\top y = Hy$. The hat matrix $H = X(X^\top X)^{-1}X^\top$ “puts the hat on $y$”: it is the orthogonal projection of $y$ onto the column space of $X$. Least squares finds the point in the space your features can reach that is closest to the data, and the residual $y - \hat y$ is perpendicular to every feature.

With one input, the slope and intercept are just covariances:

When $X^\top X$ isn’t invertible

If one column is an exact linear combination of others (say $x_2 = 3x_1$, or a full set of dummy variables plus an intercept), $X^\top X$ is singular. The fitted values $\hat y$ are still the unique projection; only the coefficients stop being unique, because there is more than one way to write the same projection.

Is a coefficient real? The t-test

Estimate the noise variance with $N - p - 1$ degrees of freedom (one is used up by each coefficient and the intercept):

If errors are uncorrelated with constant variance, $\operatorname{Var}(\hat\beta) = (X^\top X)^{-1}\sigma^2$. To test $H_0: \beta_j = 0$, standardise:

which follows a $t_{N-p-1}$ distribution under the null. $|z_j| > 2$ is the usual rule of thumb for “significant”, and $\hat\beta_j \pm 2\,\operatorname{se}(\hat\beta_j)$ is an approximate 95% interval (1.96 is the exact normal quantile).

For a single input, $\operatorname{se}(\hat\beta) \approx \dfrac{\sigma_\varepsilon}{\sqrt n\,\sigma_X}$: precision improves with more data, less noise, and more spread in the input.

Testing groups: the F-statistic

Three correlated features can each look insignificant while together they matter. Compare the full model (RSS₁, $p_1$ coefficients) with one that drops the group (RSS₀, $p_0$):

A large $F$ says the dropped group carried real information. A typical workflow: run t-tests, collect everything with $|z| < 2$, then F-test that group before deleting it.

Multiple regression is many simple ones

Regression by successive orthogonalisation (Gram–Schmidt) shows what a multiple-regression coefficient means. Regress $x_j$ on all the other inputs and keep the residual $z_j$, the part of $x_j$ nothing else explains. Then

So $\hat\beta_j$ is the effect of $x_j$ after adjusting for every other input. Econometricians call this regression anatomy (the Frisch–Waugh–Lovell theorem). It also explains multicollinearity: if $x_j$ is almost explained by the others, $z_j$ is tiny, $\langle z_j, z_j\rangle$ is small and $\operatorname{se}(\hat\beta_j)$ explodes.

The brainteasers interviewers ask

What does linear regression assume?

• The relationship is linear in the parameters (you can still use $x^2$ or $\log x$ as features).
• Errors average zero given the inputs, so there is no systematic bias. Normal errors are convenient for exact t-tests but not required.
• No perfect multicollinearity, and ideally not much of the imperfect kind.
• Homoskedasticity: the error variance doesn’t depend on the inputs. If it does, use heteroskedasticity-robust standard errors.
• Errors are uncorrelated with each other (no autocorrelation) and with the inputs.

Regress Y on X, then X on Y. Is β₁ = 1/β₂?

No. The first minimises vertical distances, the second horizontal ones. In fact $\beta_1\beta_2 = \dfrac{\operatorname{Cov}(X,Y)^2}{\operatorname{Var}(X)\operatorname{Var}(Y)} = \rho^2 \in [0, 1]$, so they are reciprocals only if the fit is perfect. Turn on “Also regress x on y” in the figure.

You duplicate every data point. What happens?

$\hat\beta$ doesn’t change ($X^\top X$ and $X^\top y$ both double, and the factors of two cancel). But the standard errors shrink by about $\sqrt 2$, so t-statistics grow. Duplicated or overlapping observations fake significance, a real danger with overlapping return windows.

Three highly correlated features: two look fine and the third has a huge standard error. What do you do?

That is multicollinearity. Options: drop the third (but it may carry unique information), combine the features (an average or a principal component), centre or transform them, or switch to ridge, lasso or principal-components regression (next two lessons).

In code

import numpy as np
rng = np.random.default_rng(8)

N, p = 200, 3
X = np.c_[np.ones(N), rng.standard_normal((N, p))]
beta_true = np.array([1.0, 2.0, 0.0, -1.0])
y = X @ beta_true + rng.standard_normal(N)

beta = np.linalg.solve(X.T @ X, X.T @ y)          # (X'X)^-1 X'y, solved stably
resid = y - X @ beta
sigma2 = resid @ resid / (N - p - 1)
se = np.sqrt(np.diag(np.linalg.inv(X.T @ X)) * sigma2)
print(np.round(beta, 3))
print(np.round(beta / se, 1))                     # t-stats: the zero coefficient has |t| < 2

# Duplicating the data: same beta, smaller standard errors
X2, y2 = np.r_[X, X], np.r_[y, y]
print(np.allclose(np.linalg.solve(X2.T @ X2, X2.T @ y2), beta))   # True