Rates of change: slope, derivative and the gradient

Before this lesson: lesson 04 (slope of a line, $e^x$ and $\log$), lesson 06 (vectors, the dot product and $\cos\theta$) and lesson 03 (NumPy). Time: about 65 minutes, plus the notebook. This is the hardest lesson of the track; take a break after section 2.

---

Why this matters #

• Training is gradient descent. Every neural network, from a logistic regression to a large language model, is trained by moving the parameters $\theta$ against the gradient of the loss: $\theta \leftarrow \theta - \eta \nabla L(\theta)$. SGD, momentum and Adam are all variants of this step.
• Backpropagation computes a gradient. It is the chain rule, applied to millions of parameters at once. PyTorch's loss.backward() fills param.grad with $\partial L / \partial \theta_j$ for every parameter.
• The learning rate is the most important hyperparameter. Too small and training is slow; too large and the loss explodes. Section 3 shows exactly where the boundary is for the simplest loss.
• Gradients answer "what if" questions. The gradient of a model output with respect to the input pixels is a saliency map. In physics, the force on an atom is minus the gradient of the energy; ML force fields for molecules are trained on such gradients.

This lesson is a first look. The Calculus track proves the rules, and the Optimization track studies gradient descent in depth.

1. Predict first #

2. The picture #

From a secant to a tangent #

The average rate of change of $f$ between $x$ and $x + h$ is the change in output divided by the change in input:

In the picture it is the slope of the secant line, the straight line through the two points $(x, f(x))$ and $(x + h, f(x + h))$ of the graph. For $f(x) = x^2$ from $x = 1$ to $x = 3$ ($h = 2$):

Now let the second point slide toward the first: $h$ gets smaller. The secant line turns, and it comes closer and closer to one line that just touches the curve at $x$: the tangent line. The slope of the tangent line is the derivative $f'(x)$.

Interactive figure: loading…

For $x^2$ at $x_0 = 1$ the secant slopes are $2 + h$ (we show this in section 3): $3.5$ for $h = 1.5$, $2.1$ for $h = 0.1$, $2.01$ for $h = 0.01$. They approach $f'(1) = 2$.

Zoom in: the curve looks like a line #

Here is a second way to see the derivative. Zoom in on a smooth curve around a point. The more you zoom, the straighter the curve looks. In the end you cannot tell it from its tangent line. So near $x_0$:

This is the local linear approximation (or linearization). Example: estimate $\sqrt{4.1}$. Take $f(x) = \sqrt x$, $x_0 = 4$, $h = 0.1$. We know $f(4) = 2$, and the slope of $\sqrt x$ at 4 is $f'(4) = \tfrac{1}{2\sqrt 4} = \tfrac14$ (from the rules in section 3). So

The true value is $2.02485\ldots$; the error is $0.00015$.

Interactive figure: loading…

The error is not just small; it is small compared with $h$. For a smooth $f$ it shrinks like $h^2$. This is the reason why the tangent is the best line through the point: any other slope $m$ leaves an error of about $|f'(x_0) - m|\,h$, which is much larger than $h^2$ for small $h$.

Many inputs: slices of a surface #

A function of two inputs, $f(x, y)$, has a graph that is a surface: the height above the point $(x, y)$ is $f(x, y)$. How steep is it? That depends on the direction. Start with the two simplest directions.

• Hold $y$ fixed and change only $x$. You walk on a slice of the surface. Its slope is the partial derivative $\partial f / \partial x$.
• Hold $x$ fixed and change only $y$. The slope of this slice is $\partial f / \partial y$.

For $f(x, y) = x^2 + 2y^2$: holding $y$ fixed, $2y^2$ is a constant, so $\partial f/\partial x = 2x$. Holding $x$ fixed, $x^2$ is a constant, so $\partial f/\partial y = 4y$.

Interactive figure: loading…

The gradient on a contour map #

A surface is hard to draw. A contour map shows the same function in 2-D: each contour line (or level set) connects the points with the same value $f(x, y) = c$. The height lines of a hiking map are contour lines.

The gradient is the vector of the partial derivatives:

It is an arrow in the input plane. On the map, it has three properties that you can test in the next figure:

1. it points in the direction in which $f$ grows fastest (steepest ascent);
2. its length $\|\nabla f\|$ is that fastest rate of growth;
3. it crosses the contour line at a right angle.

Interactive figure: loading…

The chart is a cosine curve. This is lesson 06 again: $\nabla f^\top u = \|\nabla f\|\,\|u\|\cos\theta$, and $\cos\theta$ is largest at $\theta = 0$.

Walking downhill: gradient descent #

To make $f$ small, go against the gradient. Gradient descent repeats one step:

where the learning rate $\eta > 0$ sets the step size. For $f(x) = x^2$ (one input) the gradient is $f'(x) = 2x$, so the step is $x \leftarrow x - 2\eta x = (1 - 2\eta)\,x$. Each step multiplies $x$ by the same factor $1 - 2\eta$. Everything depends on the size of this factor.

Interactive figure: loading…

3. Definitions and main results #

The derivative #

Definition. The derivative of $f$ at $x$ is the limit of the secant slopes:

If this limit exists (the same value for $h \to 0$ from both sides), $f$ is differentiable at $x$. Other notations: $\dfrac{df}{dx}$, $\dfrac{d}{dx} f(x)$; physicists write $\dot x$ for a derivative in time. "$\lim_{h \to 0}$" means: the value that the quotient approaches as $h$ gets closer and closer to 0, without being 0. The function $f'$ that maps each $x$ to $f'(x)$ is the derivative function.

Example: $f(x) = x^2$. Every algebra step:

1. $f(x + h) = (x + h)^2 = x^2 + 2xh + h^2$.
2. $f(x + h) - f(x) = x^2 + 2xh + h^2 - x^2 = 2xh + h^2$.
3. Divide by $h$ (allowed, since $h \ne 0$): $\dfrac{2xh + h^2}{h} = 2x + h$.
4. Let $h \to 0$: $2x + h \to 2x$. So $f'(x) = 2x$.

At $x = 1$: the secant slope is $2 + h$ and the derivative is $2$, as in the first figure.

Not every function has a derivative everywhere. $f(x) = |x|$ at $x = 0$: the secant slope is $|h|/h$, which is $+1$ for $h > 0$ and $-1$ for $h < 0$. The two sides disagree, so there is no limit. The graph has a corner. The ReLU activation $\max(0, x)$ has the same corner at 0; deep-learning libraries use the slope 0 there.

Local linear approximation #

Result. If $f$ is differentiable at $x$, then

Why. Divide by $h$: $\dfrac{r(h)}{h} = \dfrac{f(x + h) - f(x)}{h} - f'(x)$. The first term tends to $f'(x)$ by definition, so the difference tends to 0. $\blacksquare$

For $f(x) = x^2$ the remainder is exact and easy: $r(h) = (x + h)^2 - x^2 - 2xh = h^2$. In general, if $f$ has a second derivative, $r(h) \approx \tfrac12 f''(x)\,h^2$; this is the start of the Taylor series (Calculus lesson 3).

Rules for derivatives #

These rules follow from the definition (the Calculus track proves them). $c$ is a constant.

function	derivative	example
$c$	$0$	$\frac{d}{dx}\, 7 = 0$
$x^n$	$n\,x^{n-1}$ (power rule)	$\frac{d}{dx}\, x^3 = 3x^2$; $\frac{d}{dx}\sqrt x = \frac{d}{dx} x^{1/2} = \tfrac12 x^{-1/2}$
$e^x$	$e^x$	the slope of $e^x$ equals its height
$\log x$ ($x > 0$)	$1/x$	at $x = 1$ the slope is 1
$c\,f(x)$	$c\,f'(x)$	$\frac{d}{dx}\, 5x^2 = 10x$
$f(x) + g(x)$	$f'(x) + g'(x)$	$\frac{d}{dx}(x^2 + 3x) = 2x + 3$
$f(g(x))$	$f'(g(x))\, g'(x)$ (chain rule)	$\frac{d}{dx}\, e^{-x} = e^{-x} \cdot (-1)$

Why $e^x$ is its own derivative (the claim of lesson 04). $\dfrac{e^{x+h} - e^x}{h} = e^x \cdot \dfrac{e^h - 1}{h}$. The second factor is the secant slope of $e^x$ at 0, and it tends to 1: $(e^{0.001} - 1)/0.001 = 1.0005$. So the derivative is $e^x \cdot 1$. That the slope at 0 is exactly 1 is what makes $e$ special.

The chain rule says: if $g$ changes by a factor $g'(x)$, and $f$ changes by a factor $f'$ at $g(x)$, the two factors multiply. Example: the sigmoid $\sigma(z) = (1 + e^{-z})^{-1}$ of lesson 05. Outer function $f(u) = u^{-1}$ with $f'(u) = -u^{-2}$; inner function $u = 1 + e^{-z}$ with $u' = -e^{-z}$. So

because $1 - \sigma(z) = \dfrac{1 + e^{-z} - 1}{1 + e^{-z}} = \dfrac{e^{-z}}{1 + e^{-z}}$. At $z = 0$: $\sigma'(0) = \tfrac12 \cdot \tfrac12 = \tfrac14$; at $z = 2$: $0.8808 \cdot 0.1192 = 0.1050$. The notebook checks the formula numerically. In a network, backpropagation applies the chain rule once per layer.

What the derivative tells you #

• If $f'(x) > 0$, $f$ is increasing near $x$; if $f'(x) < 0$, it is decreasing.
• Theorem. If $f$ is differentiable and has a minimum or a maximum at an inner point $x^*$ of an interval, then $f'(x^*) = 0$.

Proof for a minimum

1. At a minimum, $f(x^* + h) - f(x^*) \ge 0$ for all small $h$.
2. For $h > 0$ the secant slope $\frac{f(x^* + h) - f(x^*)}{h}$ is $\ge 0$ (non-negative divided by positive). So its limit is $f'(x^*) \ge 0$.
3. For $h < 0$ the same quotient is $\le 0$ (non-negative divided by negative). So $f'(x^*) \le 0$.
4. Both together: $f'(x^*) = 0$. $\blacksquare$

The converse is false. $f(x) = x^3$ has $f'(0) = 0$, but 0 is neither a minimum nor a maximum: $x^3 < 0$ on the left and $> 0$ on the right. A point with $f'(x) = 0$ is a critical point (or stationary point); it may be a minimum, a maximum or neither.

Example (a log-likelihood, as in lesson 04). Three heads and one tail have the log-likelihood $\ell(p) = 3\log p + \log(1 - p)$ for $0 < p < 1$. By the rules and the chain rule, $\ell'(p) = \dfrac{3}{p} - \dfrac{1}{1 - p}$. Set it to 0: $3(1 - p) = p$, so $p = 3/4$, the fraction of heads. Since $\ell'(p) > 0$ for $p < 3/4$ and $< 0$ for $p > 3/4$, this is a maximum.

Numerical derivatives #

A computer can estimate a derivative without any rules, from two function values:

For $f(x) = x^2$ at $x = 3$ with $h = 0.1$: forward $\frac{9.61 - 9}{0.1} = 6.1$, central $\frac{9.61 - 8.41}{0.2} = 6.0$. The true value is 6. The truncation error (the error from $h > 0$) is about $\tfrac12 |f''|\,h$ for the forward difference and $\tfrac16 |f'''|\,h^2$ for the central difference. So the central difference is much more accurate for the same $h$.

But $h$ cannot be tiny. A float64 number has about 16 correct significant digits, and floats are not exact (lesson 03). For a very small $h$, $f(x + h)$ and $f(x)$ agree in almost all digits, and the subtraction loses them: this round-off error grows like $10^{-16}/h$. The total error is U-shaped in $h$. For $e^x$ at $x = 1$:

$h$	forward error	central error
$10^{-2}$	$1.4 \times 10^{-2}$	$4.5 \times 10^{-5}$
$10^{-5}$	$1.4 \times 10^{-5}$	$5.9 \times 10^{-11}$
$10^{-8}$	$6.6 \times 10^{-9}$	$6.6 \times 10^{-9}$
$10^{-11}$	$3.3 \times 10^{-5}$	$1.0 \times 10^{-5}$

Good default steps: $h \approx 10^{-8}$ for the forward difference and $h \approx 10^{-5}$ for the central difference (for values of $x$ and $f$ near 1).

Where the error terms come from

If $f$ has three derivatives, the Taylor expansion gives $f(x \pm h) = f(x) \pm f'(x)h + \tfrac12 f''(x)h^2 \pm \tfrac16 f'''(x)h^3 + \dots$

• Forward: $\frac{f(x+h) - f(x)}{h} = f'(x) + \tfrac12 f''(x)\,h + \dots$, error of order $h$.
• Central: subtract the two expansions; the even terms cancel: $\frac{f(x+h) - f(x-h)}{2h} = f'(x) + \tfrac16 f'''(x)\,h^2 + \dots$, error of order $h^2$.

Each function value has a relative round-off error of about $\varepsilon = 2.2 \times 10^{-16}$ (machine epsilon). Dividing by $h$ turns it into an error of about $\varepsilon |f| / h$. Balancing $\tfrac12|f''|h$ against $\varepsilon|f|/h$ gives the best forward step $h \approx \sqrt{\varepsilon} \approx 1.5 \times 10^{-8}$; balancing $h^2$ against $\varepsilon/h$ gives $h \approx \varepsilon^{1/3} \approx 6 \times 10^{-6}$ for the central difference. Calculus lesson 7 covers this in detail.

Partial derivatives and the gradient #

Definition. For $f : \R^n \to \R$, the partial derivative with respect to $x_j$ is the derivative in $x_j$ while all other inputs are held fixed:

where $e_j$ is the unit vector with a 1 in place $j$ and 0 elsewhere. In practice: treat every other variable as a constant and use the one-variable rules. The gradient is the vector of all partial derivatives:

It has the same size as the input $x$; it lives in the input space. The symbol $\nabla$ is read "nabla" or "del".

Example. $f(x, y) = x^2 + 3y$. Holding $y$ fixed, $3y$ is a constant: $\partial f/\partial x = 2x$. Holding $x$ fixed, $x^2$ is a constant: $\partial f/\partial y = 3$. So $\nabla f(x, y) = (2x, 3)$ and $\nabla f(1, 2) = (2, 3)$.

The direction of steepest ascent #

The partial derivatives are slopes along the axes. The slope in any other direction is a combination of them.

Definition. For a unit vector $u$ ($\|u\| = 1$), the directional derivative of $f$ at $x$ in the direction $u$ is

Theorem. If $f$ is differentiable at $x$ (for example, if all partial derivatives exist and are continuous near $x$), then

Why this is true. The linear approximation works in many variables too: $f(x + \Delta) \approx f(x) + \nabla f(x)^\top \Delta$. In 2-D, a step $\Delta = (\Delta_1, \Delta_2)$ changes $f$ by about $\frac{\partial f}{\partial x}\Delta_1$ (the $x$-part of the step) plus $\frac{\partial f}{\partial y}\Delta_2$ (the $y$-part). Put $\Delta = t\,u$ and divide by $t$. (The Calculus track gives the full proof.)

Theorem (steepest ascent). Let $\nabla f(x) \ne 0$. Among all unit vectors $u$:

1. $D_u f(x)$ is largest for $u = \nabla f(x) / \|\nabla f(x)\|$, and the largest value is $\|\nabla f(x)\|$;
2. $D_u f(x)$ is smallest for $u = -\nabla f(x) / \|\nabla f(x)\|$ (steepest descent), with value $-\|\nabla f(x)\|$;
3. $D_u f(x) = 0$ exactly for the $u$ that are orthogonal to $\nabla f(x)$.

Proof. Let $\theta$ be the angle between $\nabla f(x)$ and $u$. By the geometric formula of the dot product (lesson 06):

1. $D_u f(x) = \nabla f(x)^\top u = \|\nabla f(x)\|\,\|u\|\cos\theta = \|\nabla f(x)\| \cos\theta$, since $\|u\| = 1$.
2. $\cos\theta$ takes all values from $-1$ to $1$. It is 1 only at $\theta = 0$ ($u$ points along $\nabla f$), $-1$ only at $\theta = 180°$, and 0 only at $\theta = 90°$.
3. Multiply by the positive number $\|\nabla f(x)\|$. This gives claims 1, 2 and 3. $\blacksquare$

The gradient is perpendicular to the contour line. Along the contour line through $x$, $f$ does not change. So for the direction $u$ of the contour line (its tangent), $D_u f(x) = 0$. By claim 3, $u$ is orthogonal to $\nabla f(x)$. (This is a sketch; it needs $\nabla f(x) \ne 0$, so that the contour is a smooth curve near $x$.)

Gradient descent and the learning rate #

Gradient descent starts at $x_0$ and repeats

Why a small step decreases $f$. By the linear approximation with $\Delta = -\eta \nabla f(x)$:

If $\nabla f(x) \ne 0$, the decrease $\eta\|\nabla f(x)\|^2$ is positive. The approximation is only good for small steps, so a too large $\eta$ can increase $f$.

Exact analysis for $f(x) = x^2$. The step is $x_{k+1} = x_k - 2\eta x_k = (1 - 2\eta)\,x_k$, so

With $r = 1 - 2\eta$ and $x_0 \ne 0$, for $\eta > 0$:

learning rate	factor $r = 1 - 2\eta$	behaviour	$x_{20}$ from $x_0 = 1$
$0 < \eta < 0.5$	$0 < r < 1$	converges, same sign each step	$\eta = 0.1$: $0.8^{20} = 0.0115$
$\eta = 0.5$	$r = 0$	exact minimum after one step	$0$
$0.5 < \eta < 1$	$-1 < r < 0$	converges with a zig-zag	$\eta = 0.6$: $(-0.2)^{20} = 1.0 \times 10^{-14}$
$\eta = 1$	$r = -1$	flips between $x_0$ and $-x_0$	$1$
$\eta > 1$	$r < -1$	diverges	$\eta = 1.5$: $(-2)^{20} = 1\,048\,576$

So gradient descent on $x^2$ converges exactly when $|1 - 2\eta| < 1$, that is, $0 < \eta < 1$. The bound depends on the curvature: for $f(x) = \tfrac{a}{2} x^2$ the factor is $1 - \eta a$, and the condition is $\eta < 2/a$. A steeper bowl needs a smaller learning rate. For $x^2 + 2y^2$ (the 2-D figure) the factors are $1 - 2\eta$ for $x$ and $1 - 4\eta$ for $y$; the steep direction $y$ limits $\eta$ to below $0.5$.

Answers to section 1. (1) The average speed is $120 / 2 = 60$ km/h: the slope of the secant over two hours. The speedometer shows the instantaneous speed, the derivative at 2 pm. The car can be faster at some times and slower at others. (2) Height grows most along the gradient of the height function (steepest ascent), at the rate $\|\nabla f\|$. It does not change (to first order) along the contour line, at a right angle to the gradient. (3) With $\eta = 0.1$: $x_k = 0.8^k$, so it converges slowly to 0 ($0.0115$ after 20 steps). With $\eta = 1.5$: $x_k = (-2)^k$, it diverges.

4. Worked example #

Take $f(x, y) = x^2 + 2y^2$, the bowl of the figures, at the point $p = (1, 0.5)$.

(a) Value. $f(1, 0.5) = 1^2 + 2 \cdot 0.5^2 = 1 + 2 \cdot 0.25 = 1.5$.

(b) Partial derivatives. Hold $y$ fixed: $\partial f/\partial x = 2x = 2 \cdot 1 = 2$. Hold $x$ fixed: $\partial f/\partial y = 2 \cdot 2y = 4y = 4 \cdot 0.5 = 2$.

(c) Gradient and its length.

(d) Slopes in three directions.

• Along the $x$-axis, $u = (1, 0)$: $D_u f = 2 \cdot 1 + 2 \cdot 0 = 2$. This is $\partial f/\partial x$.
• Along $u = (1, -1)/\sqrt 2$: $D_u f = (2 \cdot 1 + 2 \cdot (-1))/\sqrt 2 = 0$. This direction is orthogonal to $\nabla f$, so it is the direction of the contour line through $p$.
• Along the gradient, $u = (2, 2)/(2\sqrt 2) = (1, 1)/\sqrt 2$: $D_u f = (2 + 2)/\sqrt 2 = 4/\sqrt 2 = 2\sqrt 2 \approx 2.828$. This is $\|\nabla f\|$, the largest possible slope.

(e) Linear approximation. Estimate $f(1.1, 0.4)$, a step $\Delta = (0.1, -0.1)$:

The true value is $1.1^2 + 2 \cdot 0.4^2 = 1.21 + 0.32 = 1.53$. The step goes along the contour line, so the linear term is 0; the change of $0.03$ comes from the curvature (the $h^2$ part).

(f) One gradient descent step with $\eta = 0.1$:

The value fell from $1.5$ to $0.82$. The linear approximation predicted $1.5 - \eta\|\nabla f\|^2 = 1.5 - 0.1 \cdot 8 = 0.7$. The true value is a little higher: the bowl curves up, and the exact correction here is $\eta^2 (2^2 + 2 \cdot 2^2) = 0.01 \cdot 12 = 0.12$, so $0.7 + 0.12 = 0.82$. ✓

5. In code #

NumPy checks the worked example. The numerical gradient uses central differences, one coordinate at a time.

import numpy as np

f = lambda v: v[0]**2 + 2 * v[1]**2
grad = lambda v: np.array([2 * v[0], 4 * v[1]])

def num_grad(f, v, h=1e-5):
    e = np.eye(len(v))                       # rows are the unit vectors e_j
    return np.array([(f(v + h * e[j]) - f(v - h * e[j])) / (2 * h) for j in range(len(v))])

p = np.array([1.0, 0.5])
g = grad(p)
print(f(p), g, np.linalg.norm(g))            # 1.5 [2. 2.] 2.8284271247461903
print(num_grad(f, p))                        # [2. 2.]

for u in [np.array([1., 0.]), np.array([1., -1.]) / np.sqrt(2), g / np.linalg.norm(g)]:
    print(g @ u)                             # 2.0, then 0.0, then 2.82842712474619

d = np.array([0.1, -0.1])
print(f(p) + g @ d, f(p + d))                # 1.5 1.5300000000000002

q = p - 0.1 * g                              # one gradient descent step
print(q, f(q))                               # [0.8 0.3] 0.8200000000000001

s = lambda x: x**2                           # forward and central difference at 3
print((s(3.1) - s(3)) / 0.1, (s(3.1) - s(2.9)) / 0.2)   # 6.100000000000012 6.000000000000005

sig = lambda z: 1 / (1 + np.exp(-z))         # sigma' = sigma (1 - sigma) at z = 2
print(sig(2) * (1 - sig(2)), (sig(2 + 1e-5) - sig(2 - 1e-5)) / 2e-5)   # 0.10499... 0.10499...

The last digits such as ...0012 are round-off (lesson 03). Compare floats with np.isclose.

6. Where it shows up #

• Training neural networks. Every optimizer (SGD, momentum, Adam, AdamW) uses $\nabla L(\theta)$, the gradient of the loss with respect to all parameters. Stochastic gradient descent estimates it on a small batch of data. See Optimization lesson 1.
• Backpropagation and automatic differentiation. PyTorch and JAX apply the chain rule through the computation, and get the full gradient for the cost of a few function evaluations. Finite differences would need $2n$ evaluations for $n$ parameters, which is impossible for $n = 10^9$.
• Gradient checking. When you write a new layer by hand, you compare its gradient with central differences on a few random inputs, as in the notebook. A relative error near $10^{-7}$ is fine; $10^{-2}$ means a bug.
• Learning-rate choice. Loss curves that explode or oscillate are the 2-D figure on a large scale: the learning rate is above the limit $2/a$ of the steepest direction. Warmup and learning-rate schedules handle this.
• Saliency and adversarial examples. The gradient of a class score with respect to the input pixels shows which pixels matter. The fast gradient sign method moves each pixel a tiny step along the sign of this gradient and can change the prediction.
• ML for science. In molecular dynamics the force on each atom is $F = -\nabla E$, minus the gradient of the potential energy. Neural network potentials learn $E$ and get the forces by automatic differentiation. Sensitivity analysis of a simulation asks for the partial derivatives of an output with respect to its parameters.

7. Common mistakes #

• Dividing by $h = 0$. The derivative is a limit as $h \to 0$, not the quotient at $h = 0$, which is $0/0$. Simplify first (as in $2x + h$), then let $h \to 0$.
• Thinking that $f'(x) = 0$ means a minimum. It can be a maximum or neither ($x^3$ at 0). In many dimensions, $\nabla f = 0$ also happens at saddle points, which are common in neural network losses.
• Drawing the gradient on the surface. $\nabla f(x)$ is a vector in the input space, with one entry per input. It points in the direction to move $x$, not up the graph.
• Going along $+\nabla f$ to minimize. The gradient points uphill. To make a loss smaller, step along $-\nabla f$.
• Choosing a tiny $h$ for numerical derivatives. $h = 10^{-15}$ gives a large round-off error. Use about $10^{-5}$ with central differences.
• Thinking that a convex bowl is safe with any learning rate. Even on $x^2$, $\eta > 1$ diverges. The limit is set by the steepest direction, not by the average one.
• Mixing up partial derivatives. For $\partial f/\partial x$ of $x^2 y$, $y$ is a constant factor: the answer is $2xy$, not $2x$.

8. Check yourself #

1. Compute the secant slope of $f(x) = x^3$ between $x = 1$ and $x = 2$, and the derivative at $x = 1$.
   Answer

   Secant: $(8 - 1)/(2 - 1) = 7$. Derivative: $f'(x) = 3x^2$, so $f'(1) = 3$. The secant slope is larger because the curve gets steeper between 1 and 2.
2. Use the linear approximation of $e^x$ at 0 to estimate $e^{0.05}$. How large is the error, roughly?
   Answer

   $e^{0.05} \approx 1 + 0.05 = 1.05$. The error is about $\tfrac12 h^2 = \tfrac12 \cdot 0.0025 = 0.00125$; the true value is $1.051271$, so the error is $0.00127$.
3. Lesson 04 claimed that $\sqrt x - \log x$ is smallest at $x = 4$. Check it with the derivative.
   Answer

   $\frac{d}{dx}(x^{1/2} - \log x) = \frac{1}{2\sqrt x} - \frac1x$. Set it to 0: $\frac{1}{2\sqrt x} = \frac1x$, so $x = 2\sqrt x$, so $\sqrt x = 2$ and $x = 4$. For $x < 4$ the derivative is negative and for $x > 4$ positive, so it is a minimum, with value $2 - \log 4 \approx 0.614$.
4. Find the gradient of $f(x, y) = 3x + xy^2$ at $(1, 2)$. What is the largest slope of $f$ at that point, and in which direction?
   Answer

   $\partial f/\partial x = 3 + y^2 = 7$, $\partial f/\partial y = 2xy = 4$. So $\nabla f(1, 2) = (7, 4)$. The largest slope is $\|\nabla f\| = \sqrt{49 + 16} = \sqrt{65} \approx 8.06$, in the direction $(7, 4)/\sqrt{65}$.
5. Give a unit direction in which $f(x, y) = x^2 + 3y$ does not change (to first order) at $(1, 2)$.
   Answer

   $\nabla f(1, 2) = (2, 3)$. A direction orthogonal to it is $(3, -2)$ (dot product $6 - 6 = 0$). As a unit vector: $(3, -2)/\sqrt{13}$, or the opposite $(-3, 2)/\sqrt{13}$.
6. Gradient descent on $f(x) = 5x^2$. Which learning rates converge?
   Answer

   $f'(x) = 10x$, so $x \leftarrow (1 - 10\eta)x$. Convergence needs $|1 - 10\eta| < 1$, that is, $0 < \eta < 0.2$. (With the rule $\eta < 2/a$ for $\tfrac a2 x^2$: $a = 10$.)

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. Define the derivative $f'(x)$ as a limit.

   Show answer

2. What is the local linear approximation of $f$ near $x$?

   Show answer

3. Estimate $\sqrt{4.1}$ with a linear approximation.

   Show answer

4. What are the derivatives of $x^n$, $e^x$ and $\log x$?

   Show answer

5. What does the partial derivative $\partial f / \partial x$ of $f(x, y)$ measure?

   Show answer

6. What is the gradient of $f : \R^n \to \R$?

   Show answer

7. Why is the gradient the direction of steepest ascent, and what is the steepest slope?

   Show answer

8. How is the gradient related to the contour line through a point?

   Show answer

9. Write the gradient descent update. For $f(x) = x^2$, which learning rates converge?

   Show answer

10. Why is the central difference better than the forward difference?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "Why do we step along the negative gradient?" — For a small step $\Delta$, $f(x + \Delta) \approx f(x) + \nabla f^\top \Delta$. Among steps of a fixed length, $\nabla f^\top \Delta$ is most negative when $\Delta$ points along $-\nabla f$ (Cauchy–Schwarz, or $\cos\theta = -1$). So $-\nabla f$ gives the fastest first-order decrease.
• "What happens if the learning rate is too large?" — The steps overshoot. On $f(x) = \frac a2 x^2$ the update multiplies $x$ by $1 - \eta a$; for $\eta > 2/a$ this has size above 1 and the iterates diverge. In practice the loss oscillates or becomes nan.
• "How do you check that your backward pass is correct?" — Gradient checking: compare the analytic gradient with central differences $\frac{f(x + he_j) - f(x - he_j)}{2h}$, $h \approx 10^{-5}$, in float64, using the relative error.
• "Why not train with finite differences?" — They need $2n$ function evaluations per gradient, one pair per parameter, and they are less accurate. Reverse-mode automatic differentiation (backpropagation) gets all $n$ partial derivatives for a small constant times the cost of one evaluation.
• "The gradient is zero. Are you at a minimum?" — Not necessarily: it can be a maximum or a saddle point. In high dimensions, saddle points are much more common than poor local minima (Dauphin et al., 2014).

10. Practice #

The notebook for this lesson runs in Colab or in your browser. Five exercises, each with a check cell and a folded solution:

1. Write the forward and central differences, and plot their errors against $h$ for $e^x$ at 1 (the U-shaped curves).
2. Check $\sigma'(z) = \sigma(z)(1 - \sigma(z))$ numerically on 201 points.
3. Write a numerical gradient for any $f : \R^n \to \R$ and compare it with analytic gradients.
4. Run gradient descent on $x^2 + 5y^2$, plot the path on the contours, and find the largest stable learning rate.
5. Show that the error of the linear approximation of $e^x$ at 0, divided by $h^2$, tends to $1/2$.

11. Watch and read #

The essence of calculus3Blue1Brown · Essence of calculus, chapter 1 · 17 min

Why calculus thinks in tiny changes; a good first view before the formulas.

The paradox of the derivative | Chapter 2, Essence of calculus3Blue1Brown · Essence of calculus, chapter 2 · 17 min

The derivative as the limit of $\Delta f / \Delta x$, as in the secant figure, and the derivative of $x^2$ and $x^3$.

Why the gradient is the direction of steepest ascentKhan Academy · Multivariable calculus · 11 min

The same argument as section 3: the directional derivative is a dot product with the gradient.

Gradient descent, how neural networks learn | Deep Learning Chapter 23Blue1Brown · Neural networks, chapter 2 · 21 min

Gradient descent on the loss of a real network; the gradient as "which weights matter most".

The spelled-out intro to neural networks and backpropagation: building microgradAndrej Karpathy · 2 h 26 min (watch 8:08 to 19:09)

Derivatives of a simple function, computed numerically with a small h, then with several inputs (partial derivatives).

• Deisenroth, Faisal and Ong, Mathematics for Machine Learning, free PDF: §5.1 (differentiation of univariate functions) and §5.2 (partial differentiation and gradients).
• OpenStax, Calculus Volume 1, free: chapter 3 (derivatives), sections 3.1–3.3 and 3.6 (the chain rule).
• Next steps: Calculus lesson 1 (derivatives in one variable, with proofs) and Optimization lesson 1 (loss surfaces and gradient descent).