Gradient descent
Find the bottom of a loss by repeatedly stepping downhill. Each step moves the coefficients against the gradient, scaled by the learning rate.
01The formula
- the coefficients after k steps
- the learning rate: how far to move on each step
- the gradient: the direction in which the loss rises fastest
- the loss to minimise, here a squared error
02See it move
03Where it is used
- Quant researcherTrains neural nets and other large models on market data, where there is no closed-form answer.
- Market makerUpdates model parameters online, one small gradient step per batch of new trades.
- Portfolio managerSolves large portfolio optimisations with constraints by projected gradient steps.
04Why traders use it
Most modern models are fitted this way. Knowing how the learning rate and the shape of the loss interact explains why training stalls, zig-zags or blows up.
05Worked example
On a bowl stretched by correlation 0.70, starting with loss 3.250, 15 steps with η = 0.15 bring the loss down to 0.1417, still 0.687 from the minimum. Any η above 0.588 would diverge.
06How it is built
- Here the loss is , with gradient .
- One step multiplies the error by . Along each eigen-direction of the error shrinks by .
- Steps converge only if for every , so ; for this bowl .
- The flattest direction, , sets how slowly the last part of the error disappears.
07The common mistake
Turning the learning rate up to go faster. Past the stability limit every step overshoots by more than the last and the loss explodes; on a long, narrow valley the safe rate is set by the steepest wall.
08In Python and LaTeX
import numpy as np
rho, eta, steps = 0.7, 0.15, 15
S = np.array([[1, rho], [rho, 1]])
star, b = np.array([1.0, 0.0]), np.array([-1.5, 1.5])
for _ in range(steps):
b = b - eta * 2 * S @ (b - star)
d = b - star
print(round(d @ S @ d, 4)) # loss after the last step\boldsymbol\beta_{k+1} = \boldsymbol\beta_k - \eta\,\nabla L(\boldsymbol\beta_k)09Where it fits
Rather not code the maths yourself? Describe your trading idea and let AI build the bot.
Build a bot with AgenKitThe worked example and the model run the same code, so every number matches. Models simplify markets on purpose. This is education, not financial advice.