Skip to content
LearnMLByDoing
Free

2.1 Gradient descent, saddle points and optimizer dynamics

40 minFree lessonintermediateMLCCDLAdamMML

An optimizer turns the local slope of the loss into a step, and SGD, momentum and Adam turn the same slope into very different paths across a bowl, a saddle and a curved valley.

01The loss is a landscape

Training a model means choosing parameters θ\theta that make a scalar loss L(θ)\mathcal{L}(\theta) small. With two parameters the loss is a surface: θ1\theta_1 and θ2\theta_2 span the floor and L\mathcal{L} is the height. Every optimizer in this lesson sees only one thing about that surface at a time: its slope at the current point.

θ (parameters)[]2 values

Two parameters so the loss can be drawn. A real network has millions, and every update below is applied elementwise to that whole vector.

The slope is the gradient ∇L(θ)=(∂L/∂θ1, ∂L/∂θ2)\nabla\mathcal{L}(\theta) = (\partial\mathcal{L}/\partial\theta_1,\ \partial\mathcal{L}/\partial\theta_2), the direction of steepest ascent. Near θ\theta the surface is well approximated by its tangent plane, the grey patch under the emerald ball:

L(θ+δ)  ≈  L(θ)+∇L(θ)⊤δ\tok{loss}{\mathcal{L}}(\theta+\delta)\;\approx\;\tok{loss}{\mathcal{L}}(\theta)+\tok{gradient}{\nabla\mathcal{L}(\theta)}^{\top}\delta
First-order Taylor model. The amber arrow is −∇L, the steepest way down the plane.

The surface here is the convex bowl L(θ)=12θ12+2θ22\mathcal{L}(\theta)=\tfrac12\theta_1^2+2\theta_2^2. Its gradient is ∇L=(θ1, 4θ2)\nabla\mathcal{L}=(\theta_1,\ 4\theta_2), so at the start (−1.6, 1.4)(-1.6,\ 1.4) it is (−1.6, 5.6)(-1.6,\ 5.6), and the steepest way down, −∇L=(1.6, −5.6)-\nabla\mathcal{L}=(1.6,\ -5.6), points mostly along θ2\theta_2 rather than at the minimum at the origin. The bowl is four times steeper along θ2\theta_2 than along θ1\theta_1, and that mismatch is what the rest of the lesson is about.

Three optimizers start from (−1.6, 1.4)(-1.6,\ 1.4): SGD in emerald, momentum in amber and Adam in cyan. Each dot is one step, so dot spacing is step size. The emerald ball is labelled SGD, as the plain update is in deep-learning libraries, but with the cockpit's gradient noise at 0 it uses the exact gradient: it is full-batch gradient descent, with no minibatch noise, so every run is deterministic. Step 3 turns the noise on.

Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Ch. 5.1 and 5.8

Deep Learning, Goodfellow, Bengio & Courville, Ch. 4.3

02Gradient descent and the learning rate

Gradient descent moves a small distance against the gradient. The learning rate η\eta sets how far:

θt+1=θt−η ∇L(θt)\theta_{t+1}=\theta_t-\tok{learning_rate}{\eta}\,\tok{gradient}{\nabla\mathcal{L}(\theta_t)}
Plain gradient descent, the emerald ball.

The viewport is at η=0.42\eta = 0.42, so the first step from (−1.6, 1.4)(-1.6,\ 1.4) is −0.42⋅(−1.6, 5.6)=(0.672, −2.352)-0.42\cdot(-1.6,\ 5.6) = (0.672,\ -2.352), landing at (−0.928, −0.952)(-0.928,\ -0.952): θ1\theta_1 moved a little toward 0, but θ2\theta_2 jumped right past 0 to the other side. To see when this converges, the derivation below needs one more object. The Hessian HH is the table of second derivatives, how fast the slope itself changes. For this bowl it is diagonal: ∂2L/∂θ12=1\partial^2\mathcal{L}/\partial\theta_1^2 = 1 and ∂2L/∂θ22=4\partial^2\mathcal{L}/\partial\theta_2^2 = 4, the curvature along each axis. Its eigenvalues λi\lambda_i are those curvatures, 1 and 4, and its eigen-directions are the two axes.

When does it converge on a quadratic?1/4
  1. 1
    L(θ−ηg)≈L(θ)−η g⊤g=L(θ)−η∥g∥2\mathcal{L}(\theta-\eta g)\approx\mathcal{L}(\theta)-\eta\,g^{\top}g=\mathcal{L}(\theta)-\eta\|g\|^2

    Put δ = −ηg into the Taylor model. To first order the loss can only fall, but the model is only trusted for small η.

Because HH is diagonal, the bowl is two separate one-dimensional problems, one per axis, each a parabola 12λθ2\tfrac12\lambda\theta^2. The figure shows one of them. Every step multiplies θ\theta by the same factor 1−ηλ1-\eta\lambda, so the iterates hop across the parabola: on the same side while the factor is positive, from side to side when it is negative, and further out each time once ∣1−ηλ∣>1|1-\eta\lambda|>1. Its learning-rate slider is the cockpit's η\eta, so the 3D race follows it.

Loading figure
Drag θ0\theta_0 or the marker on the ηλ\eta\lambda strip. With λ=4\lambda = 4 (the θ2\theta_2 axis) the zones change at η=0.25\eta = 0.25 and η=0.5\eta = 0.5; with λ=1\lambda = 1 (the θ1\theta_1 axis) every η\eta on the slider stays in the monotone zone.

Set η=0.25=1/λmax⁡\eta = 0.25 = 1/\lambda_{\max}, the learning rate that makes the θ2\theta_2 factor exactly 1−4⋅0.25=01 - 4\cdot0.25 = 0.

Set η=0.5=2/λmax⁡\eta = 0.5 = 2/\lambda_{\max}, exactly the stability limit.

Cross the stability limit: set η=0.52\eta = 0.52, just above 2/λmax⁡=0.52/\lambda_{\max} = 0.5, on the same bowl and start.

The slow direction sets the pace. Along θ1\theta_1 the factor is 1−η1-\eta, so a learning rate small enough for the steep axis makes the shallow axis crawl. The ratio κ=λmax⁡/λmin⁡\kappa=\lambda_{\max}/\lambda_{\min}, the condition number, measures how bad this trade-off is; it is 4 here and can be in the thousands for real networks.

Deep Learning, Goodfellow, Bengio & Courville, Ch. 4.3.1

Machine Learning Crash Course, Google, Gradient descent; Learning rate

03Saddle points: zero gradient, not a minimum

A critical point has ∇L=0\nabla\mathcal{L}=0. The Hessian decides what it is: all eigenvalues positive is a minimum, mixed signs is a saddle. The surface now is L=θ12−θ22+12θ24\mathcal{L}=\theta_1^2-\theta_2^2+\tfrac12\theta_2^4, with Hessian diag⁡(2,−2)\operatorname{diag}(2,-2) at the origin and two minima at (0,±1)(0,\pm1) where L=−0.5\mathcal{L}=-0.5. All three optimizers start at θ2=10−3\theta_2=10^{-3}, almost exactly on the ridge.

θ2,t+1=θ2,t−η (−2θ2,t+2θ2,t3)  ≈  (1+2η) θ2,t\theta_{2,t+1}=\theta_{2,t}-\tok{learning_rate}{\eta}\,(-2\theta_{2,t}+2\theta_{2,t}^{3})\;\approx\;(1+2\eta)\,\theta_{2,t}
Near the saddle the unstable coordinate grows by only 1 + 2η per step.

With η=0.04\eta=0.04 SGD needs about ln⁡(500)/ln⁡(1.08)≈81\ln(500)/\ln(1.08)\approx 81 steps to grow θ2\theta_2 from 10−310^{-3} to 0.50.5. Meanwhile the tangent plane is nearly flat, the HUD shows ∥∇L∥\|\nabla\mathcal{L}\| close to zero, and the loss sits at 00, well above the minimum of −0.5-0.5. A small gradient at a loss that is not the best you have seen is the signature of a saddle, not of convergence.

In high-dimensional networks, critical points with high loss are overwhelmingly saddles rather than local minima, so long plateaus are a more common failure than being truly trapped. Minibatch noise and the adaptive methods below are the practical remedies. (On this surface and on the Rosenbrock valley the drawn height is log-compressed so the low region stays visible; the HUD values are the true L\mathcal{L}.)

Start at θ2=0\theta_2 = 0 exactly instead of 10−310^{-3}.

Real training does not use exact gradients. A minibatch of BB examples gives an estimate whose noise shrinks roughly like 1/B1/\sqrt{B}, so small batches mean noisy steps. The cockpit's gradient noise slider adds seeded Gaussian noise of that kind to every optimizer's gradient (the same seed on every restart, so a run repeats exactly). Above 0 the emerald ball really is stochastic gradient descent.

Same ridge start, now with gradient noise σ=0.2\sigma = 0.2.

Deep Learning, Goodfellow, Bengio & Courville, Ch. 8.2.3

04Momentum: velocity across a ravine

The surface is now a scaled Rosenbrock function, L=0.1 [(1−θ1)2+5(θ2−θ12)2]\mathcal{L}=0.1\,[(1-\theta_1)^2+5(\theta_2-\theta_1^2)^2], a curved valley with its minimum at (1,1)(1,1). At the minimum the Hessian eigenvalues are about 5.165.16 across the valley and 0.0390.039 along it, a condition number of about 133. SGD drops into the valley and then crawls along its floor.

ut+1=β ut−η ∇L(θt),θt+1=θt+ut+1\tok{velocity}{u}_{t+1}=\beta\,\tok{velocity}{u}_t-\tok{learning_rate}{\eta}\,\tok{gradient}{\nabla\mathcal{L}(\theta_t)},\qquad \theta_{t+1}=\theta_t+\tok{velocity}{u}_{t+1}
Heavy-ball momentum, the amber ball. We write the velocity as u to keep v free for Adam.
What the velocity remembers1/3
  1. 1
    ut=−η∑k=0t−1βk ∇L(θt−1−k)u_t=-\eta\sum_{k=0}^{t-1}\beta^{k}\,\nabla\mathcal{L}(\theta_{t-1-k})

    Unrolled, the velocity is an exponentially weighted sum of past gradients.

Both effects help on the valley: the cross-valley gradient alternates and is averaged away, while the small, consistent along-valley gradient is amplified. Hover the velocity symbol uu to see the amber velocity arrow, drawn three times longer than the actual step. (The valley's drawn height is log-compressed so its floor stays visible; the HUD shows the true L\mathcal{L}.)

The sum ∑kβkgt−k\sum_k\beta^k g_{t-k} is an exponential moving average in disguise. Write mt=βmt−1+(1−β)gtm_t=\beta m_{t-1}+(1-\beta)g_t, starting from m0=0m_0=0; unrolling gives mt=(1−β)∑kβkgt−km_t=(1-\beta)\sum_k\beta^k g_{t-k}, so the velocity is ut=−η mt/(1−β)u_t=-\eta\,m_t/(1-\beta). The figure feeds a noisy gradient with mean 1 into that average. Its β\beta slider is the cockpit's momentum.

Loading figure
Amber bars are the noisy gradients, the cyan line is their moving average mtm_t. Larger β\beta smooths more and lags more, and the gain ∑βk\sum\beta^k that multiplies the step grows toward 1/(1−β)1/(1-\beta): 10 at β=0.9\beta = 0.9, reached only after a few dozen steps.
Loading figure
The race as loss curves, computed with the viewport's update rules. On the valley SGD is still above the dashed line after 600 steps, while momentum (136) and Adam (129) cross it.

Move to the bowl with a gentler momentum, β=0.5\beta = 0.5.

Deep Learning, Goodfellow, Bengio & Courville, Ch. 8.3.2

05Adam: per-coordinate step sizes

Adam keeps two exponential moving averages per parameter: one of the gradient, which estimates the direction, and one of the squared gradient, which estimates the scale. It then takes a step of roughly η\eta in every coordinate, shrinking it where the gradient is noisy or changes sign.

mt=β1mt−1+(1−β1) gt,vt=β2vt−1+(1−β2) gt2m_t=\beta_1 m_{t-1}+(1-\beta_1)\,g_t,\qquad v_t=\beta_2 v_{t-1}+(1-\beta_2)\,g_t^{2}
g_t = ∇L(θ_{t−1}); the square is elementwise.
m^t=mt1−β1t,v^t=vt1−β2t,θt=θt−1−η m^tv^t+ϵ\tok{first_moment}{\hat m_t}=\frac{m_t}{1-\beta_1^{t}},\quad \tok{second_moment}{\hat v_t}=\frac{v_t}{1-\beta_2^{t}},\quad \theta_t=\theta_{t-1}-\tok{learning_rate}{\eta}\,\frac{\tok{first_moment}{\hat m_t}}{\sqrt{\tok{second_moment}{\hat v_t}}+\epsilon}
Bias-corrected moments and the Adam update, the cyan ball.
Why divide by 1 − βᵗ1/3
  1. 1
    mt=(1−β1)∑k=1tβ1 t−k gkm_t=(1-\beta_1)\sum_{k=1}^{t}\beta_1^{\,t-k}\,g_k

    m starts at 0, so it is a weighted sum whose weights add up to less than 1.

Three steps with numbers make this concrete. Take a constant gradient g=2g = 2 with the default β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999:

tmtvtm^tv^tm^t/v^tmt/vt10.2000.004002413.1620.3800.008002414.2530.5420.011992414.95\begin{array}{c|cc|cc|c|c} t & m_t & v_t & \hat m_t & \hat v_t & \hat m_t/\sqrt{\hat v_t} & m_t/\sqrt{v_t} \\ \hline 1 & 0.200 & 0.00400 & 2 & 4 & 1 & 3.16 \\ 2 & 0.380 & 0.00800 & 2 & 4 & 1 & 4.25 \\ 3 & 0.542 & 0.01199 & 2 & 4 & 1 & 4.95 \end{array}
The raw moments start near 0 and creep up; the corrected ones are exactly g = 2 and g² = 4 from the first step, so every step is exactly η. Without correction the step would be 3 to 5 times η and still drifting.

The figure runs the same averages on a noisy gradient. Its β1\beta_1 is the cockpit's, so the cyan ball in the viewport uses the value you set here.

Loading figure
Cyan is the raw average mtm_t, which starts near 0; dashed emerald is m^t=mt/(1−β1t)\hat m_t = m_t/(1-\beta_1^t), which starts at g1g_1; dashed white is v^t\sqrt{\hat v_t}, the scale Adam divides by. Switch bias correction off and the first step jumps from η\eta to 3.16 η3.16\,\eta.
m, v (Adam state)[,]4 values

One first and one second moment per parameter, the same shape as θ. For a large model, Adam's state is two extra copies of the weights.

Adam is two older ideas combined: the first moment is momentum, and dividing by v^\sqrt{\hat v} is RMSProp. Setting β1=0\beta_1 = 0 removes the momentum part, so Adam becomes RMSProp with bias correction on vv.

Set Adam's β1=0\beta_1 = 0 on the bowl.

Adam: A Method for Stochastic Optimization, Kingma & Ba, 2015, Sec. 2 and 3

Deep Learning, Goodfellow, Bengio & Courville, Ch. 8.5.3

06Reading a trajectory

Loss curves tell you that something is wrong; trajectories tell you what. Three signatures cover most problems:

Divergence. Steps alternate sides of the valley and get longer, the dots spread apart, and L\mathcal{L} rises. The cause is ∣1−ηλmax⁡∣>1|1-\eta\lambda_{\max}|>1. Lower η\eta below 2/λmax⁡2/\lambda_{\max}.

Saddle stall. Dots bunch up, ∥∇L∥\|\nabla\mathcal{L}\| is close to zero, yet L\mathcal{L} is above the best value, and then the path suddenly peels away. Use momentum, an adaptive method, or minibatch noise.

Ill-conditioned crawl. Steady, tiny steps along a curved valley floor; the loss falls slowly but never rises. Use momentum or per-coordinate scaling, or reparameterize so the curvature is more even.

Loading figure
The same race as loss curves. A diverging run is the curve that climbs and ends in a rose ring; a converging one falls below the dashed line. In the viewport's state, SGD leaves the domain at step 12 while momentum (147) and Adam (116) converge.

Schedules, clipping and AdamW. Three standard tools sit next to "lower η\eta". A learning-rate schedule changes η\eta during training: a short warm-up from a small value lets Adam's v^\hat v estimates settle before full-size steps, and a step or cosine decay at the end shrinks the steps so the noisy jitter around the minimum dies down (turn on gradient noise in the cockpit to see that jitter). Gradient clipping rescales the gradient to norm cc whenever ∥g∥>c\|g\|>c: the direction is kept, but one exploding gradient can no longer throw the parameters off the map. AdamW applies weight decay directly to the weights, θ←θ−η (m^/(v^+ϵ)+λθ)\theta\leftarrow\theta-\eta\,(\hat m/(\sqrt{\hat v}+\epsilon)+\lambda\theta), instead of adding λ∥θ∥2\lambda\|\theta\|^2 to the loss, where Adam's division by v^\sqrt{\hat v} would weaken the penalty on weights with large gradients. Lesson 2.3 covers regularisation itself.

Deep Learning, Goodfellow, Bengio & Courville, Ch. 8.2

Compiling scene
Loading editor

Questions learners ask

What is the difference between gradient descent and stochastic gradient descent?
Gradient descent uses the exact gradient of the full training loss. Stochastic gradient descent estimates it from a random minibatch, which is much cheaper per step and adds noise. The update rule is the same. In this lesson the runs are exact gradient descent until you raise the gradient-noise slider, which adds minibatch-like noise; that noise is also what helps SGD leave saddle points.
How do I choose a learning rate?
On a quadratic, gradient descent is stable only when the learning rate is below 2 divided by the largest Hessian eigenvalue, and the smallest eigenvalue then sets how slowly it converges. In practice, sweep the learning rate on a log scale for a short run, pick the largest value whose loss falls smoothly, and add warm-up and decay. For Adam on neural networks, 0.001 is a common starting point.
Why does Adam need bias correction?
Adam's moment estimates start at zero, so during the first steps they are shrunk by a factor of 1 minus beta to the power t. Dividing by that factor makes the early steps the intended size. Without it, the first step with the default betas is about 3.16 times the learning rate.

Next in Classical Machine Learning

2.2Linear and logistic regression from first principlesClosed-form least squares as a projection, then the sigmoid, cross-entropy and the convex surface it creates.

Sources

  • Machine Learning Crash Course, Google
  • Deep Learning, Goodfellow, Bengio & Courville
  • Adam: A Method for Stochastic Optimization, Kingma & Ba, 2015
  • Mathematics for Machine Learning, Deisenroth, Faisal & Ong