Derivatives in one variable: slope, linear approximation and the rules

Before this lesson: nothing from this track; this is the first lesson. You need functions, $e^x$, $\log x$ and sums (foundations). You may have seen derivatives in school; we review them from the start, with the view that the rest of the track needs. Time: about 60 minutes, plus the notebook.

---

Why this matters #

Training a model means changing its parameters so that a loss goes down. The derivative says which way is down and how steep it is. Everything in this track is built on the one-variable ideas of this lesson:

• Backpropagation is the chain rule, applied many times (lessons 04–06).
• Activation functions are chosen partly for their derivatives. The sigmoid has $\sigma'(x) \le 1/4$, so long chains of sigmoids give tiny gradients (vanishing gradients). ReLU has derivative 1 where it is active, but no derivative at 0.
• Learning rates are limited by curvature: on a quadratic with curvature $a$, gradient descent diverges for any step size above $2/a$. The same rule, with the largest Hessian eigenvalue in place of $a$, holds in many dimensions (lesson 03).
• Taylor polynomials are behind Newton's method (lesson 03), numerical differentiation and gradient checks (lesson 07), and the Laplace approximation (lesson 11).

1. Predict first #

2. The picture #

Slope of a straight line. A line $y = mx + c$ has slope $m$: move 1 step to the right and $y$ changes by $m$. For a curve, the slope is different at each point. We find it in two steps.

Step 1: the secant. Take two points on the graph, at $x$ and at $x + h$. The line through them is a secant line. Its slope is "rise over run", the difference quotient

Step 2: let $h$ go to 0. For $f(x) = x^2$ at $x = 1$, the difference quotient is $\big((1+h)^2 - 1\big)/h = (2h + h^2)/h = 2 + h$. With $h = 0.1, 0.01, 0.001$ it is $2.1$, $2.01$, $2.001$. The secant turns into the tangent line, and its slope goes to $2$. This limit is the derivative: $f'(1) = 2$.

Interactive figure: loading…

Zoom in. A second way to see the derivative: zoom in on the graph around a point. A smooth curve looks more and more like a straight line, and that line is the tangent. The vertical gap between the curve and the tangent at distance $h$ is the error

It is not enough that $r(h)$ goes to 0 (that is true for every continuous $f$ and any line through the point). The important fact is that $r(h)$ goes to 0 faster than $h$: $r(h)/h \to 0$. In the zoomed picture, where $h$ is always "one window wide", the gap disappears.

Interactive figure: loading…

3. Definitions and main results #

The derivative #

Definition. $f$ is differentiable at $x$ if the limit

exists (as a finite number). The number $f'(x)$ is the derivative of $f$ at $x$. Other notations: $\dfrac{df}{dx}$, $\dfrac{d}{dx} f(x)$, $Df(x)$. The limit must be the same when $h$ goes to 0 from the right ($h > 0$) and from the left ($h < 0$).

Linear approximation and $o(h)$ #

Theorem (derivative = best linear approximation). $f$ is differentiable at $x$ exactly when there is a number $m$ such that

Then $m = f'(x)$.

Proof.

1. Define $r(h) = f(x+h) - f(x) - m h$. Divide by $h$: $\dfrac{r(h)}{h} = \dfrac{f(x+h) - f(x)}{h} - m$.
2. So $r(h)/h \to 0$ holds exactly when the difference quotient tends to $m$, that is, when $f'(x)$ exists and equals $m$. $\blacksquare$

We write $r(h) = o(h)$ (read "little-o of $h$") for "$r(h)/h \to 0$". In plain words: the error is small even compared with the small step $h$. Examples: $h^2$ and $h^3$ are $o(h)$; $0.5h$ and $\sqrt{|h|}$ are not.

For $f(x) = x^2$: $(x+h)^2 = x^2 + 2x\,h + h^2$, so $f'(x) = 2x$ and $r(h) = h^2$. At $x = 1$ with $h = 0.1$: $r = 0.01$ and $r/h = 0.1$. With $h = 0.01$: $r = 0.0001$ and $r/h = 0.01$.

This is the form of the derivative that generalizes. In lesson 02 the number $m$ becomes the gradient vector, and in lesson 03 it becomes the Jacobian matrix.

One consequence: a differentiable function is continuous at $x$, because $f(x+h) - f(x) = f'(x)h + r(h) \to 0$.

Kinks: where the derivative does not exist #

For $f(x) = |x|$ at $x = 0$, the difference quotient is $|h|/h$. It is $+1$ for every $h > 0$ and $-1$ for every $h < 0$. The one-sided derivatives (limits from the right and from the left) exist but differ, so $f'(0)$ does not exist. For $\mathrm{ReLU}(x) = \max(0, x)$ at 0 they are 1 and 0.

Neural networks use ReLU anyway. Two reasons:

• The kink is a single point. A pre-activation is exactly $0.0$ only rarely, and everywhere else the derivative is 1 or 0.
• At the kink, frameworks return a fixed value: PyTorch and JAX use $\mathrm{ReLU}'(0) = 0$. This choice is a subgradient: for a convex $f$, a number $g$ with $f(y) \ge f(x) + g\,(y - x)$ for all $y$ (a line through the point that stays below the graph). For ReLU at 0, every $g \in [0, 1]$ is a subgradient. The optimization track builds methods on subgradients.

The rules #

Let $f$ and $g$ be differentiable at $x$ and let $c$ be a number.

Rule	Formula
linearity	$(f + g)' = f' + g'$, $\ (cf)' = c f'$
product	$(fg)' = f'g + fg'$
quotient	$(f/g)' = (f'g - fg')/g^2$, where $g(x) \ne 0$
chain	$\big(f(g(x))\big)' = f'(g(x))\, g'(x)$

Why the chain rule multiplies. Let $u = g(x)$ and $y = f(u)$. A small change $\Delta x$ changes $u$ by about $g'(x)\,\Delta x$. That change of $u$ changes $y$ by about $f'(u)$ times itself. So $\Delta y \approx f'(u)\, g'(x)\, \Delta x$: rates of change multiply. Example: $y = (3x + 1)^2$ at $x = 1$. Here $u = 3x + 1 = 4$, $dy/du = 2u = 8$ and $du/dx = 3$, so $dy/dx = 8 \cdot 3 = 24$. Check by expanding: $y = 9x^2 + 6x + 1$, $y' = 18x + 6 = 24$ at $x = 1$. ✓

Proofs of the product rule and the chain rule

Product rule. Write $f(x+h) = f + f'h + r_1(h)$ and $g(x+h) = g + g'h + r_2(h)$, with all values at $x$ and $r_1, r_2 = o(h)$. Multiply: $f(x+h)g(x+h) = fg + (f'g + fg')h + \big[f'g'h^2 + (f + f'h) r_2 + (g + g'h) r_1 + r_1 r_2\big]$. Every term in the bracket, divided by $h$, goes to 0. By the linear-approximation theorem, $(fg)' = f'g + fg'$.

Chain rule. Let $u = g(x)$ and $k(h) = g(x+h) - g(x)$. Then $k(h) = g'(x)h + r_g(h)$ and $k(h) \to 0$.

1. For $f$ at $u$, write $f(u + k) - f(u) = f'(u)k + \rho(k)\,k$, where $\rho(k) = r_f(k)/k$ for $k \ne 0$ and $\rho(0) = 0$. Then $\rho(k) \to 0$ as $k \to 0$.
2. So $f(g(x+h)) - f(g(x)) = f'(u)\,k(h) + \rho(k(h))\,k(h)$.
3. Divide by $h$: $\dfrac{k(h)}{h} \to g'(x)$, and $\rho(k(h)) \to 0$ because $k(h) \to 0$.
4. The limit is $f'(u)\,g'(x)$. $\blacksquare$

Lesson 04 states the chain rule for many variables: the rates become Jacobian matrices, and "multiply" becomes matrix multiplication.

The derivatives that ML uses #

$f(x)$	$f'(x)$	where it appears
$x^n$	$n x^{n-1}$	polynomials, squared error
$e^x$	$e^x$	softmax, exponential families
$\log x$ ($x > 0$)	$1/x$	log-likelihood, cross-entropy
$\sin x$, $\cos x$	$\cos x$, $-\sin x$	positional encodings, Fourier features
sigmoid $\sigma(x) = \dfrac{1}{1 + e^{-x}}$	$\sigma(x)\big(1 - \sigma(x)\big)$	logistic regression, gates in LSTMs
$\tanh x$	$1 - \tanh^2 x$	RNNs, older MLPs
softplus $\log(1 + e^x)$	$\sigma(x)$	smooth ReLU, positive parameters
$\log \sigma(x)$	$1 - \sigma(x) = \sigma(-x)$	logistic loss, GAN losses
ReLU $\max(0, x)$	$1$ if $x > 0$, $0$ if $x < 0$	almost every modern network
GELU $x\,\Phi(x)$	$\Phi(x) + x\,\varphi(x)$	transformers

Here $\Phi$ and $\varphi$ are the cumulative distribution function and the density of the standard normal distribution.

Derivation of $\sigma'$, step by step.

1. Write $\sigma(x) = (1 + e^{-x})^{-1}$: an outer function $u^{-1}$ and an inner function $u = 1 + e^{-x}$.
2. Outer derivative: $\dfrac{d}{du} u^{-1} = -u^{-2}$. Inner derivative: $\dfrac{du}{dx} = -e^{-x}$ (chain rule again, for $e^{-x}$).
3. Multiply: $\sigma'(x) = -(1 + e^{-x})^{-2} \cdot (-e^{-x}) = \dfrac{e^{-x}}{(1 + e^{-x})^2}$.
4. Split the fraction: $\sigma'(x) = \dfrac{1}{1 + e^{-x}} \cdot \dfrac{e^{-x}}{1 + e^{-x}}$.
5. The first factor is $\sigma(x)$. The second is $\dfrac{(1 + e^{-x}) - 1}{1 + e^{-x}} = 1 - \sigma(x)$.
6. So $\sigma'(x) = \sigma(x)\big(1 - \sigma(x)\big)$.

The product $s(1 - s)$ of two numbers that add up to 1 is largest when $s = 1/2$, that is, at $x = 0$. So $\sigma'(x) \le \sigma'(0) = 1/4$.

The same method gives the others. Softplus: $\dfrac{d}{dx}\log(1 + e^x) = \dfrac{e^x}{1 + e^x} = \sigma(x)$. Log-sigmoid: $\dfrac{d}{dx}\log\sigma(x) = \dfrac{\sigma'(x)}{\sigma(x)} = 1 - \sigma(x)$. Tanh, by the quotient rule on $\sinh/\cosh$: $\tanh' = (\cosh^2 - \sinh^2)/\cosh^2 = 1 - \tanh^2$.

Second derivative, curvature and convexity #

The second derivative $f''(x)$ is the derivative of $f'$. It says how the slope changes.

• $f'' > 0$: the slope increases, the graph bends upwards (like $x^2$).
• $f'' < 0$: the slope decreases, the graph bends downwards (like $-x^2$).

A function is convex on an interval if every chord lies above the graph: $f\big(\lambda a + (1-\lambda) b\big) \le \lambda f(a) + (1 - \lambda) f(b)$ for all $a, b$ in the interval and $\lambda \in [0, 1]$.

Theorem. If $f$ is twice differentiable on an open interval $I$, then $f$ is convex on $I$ exactly when $f''(x) \ge 0$ for all $x \in I$.

Examples: $x^2$, $e^x$, $-\log x$ ($f'' = 1/x^2$) and softplus ($f'' = \sigma' > 0$) are convex. The sigmoid is not: $\sigma'' = \sigma(1 - \sigma)(1 - 2\sigma)$ is positive for $x < 0$ and negative for $x > 0$. Note: $f'' > 0$ everywhere implies strictly convex, but the converse is false. $x^4$ is strictly convex, yet $f''(0) = 0$.

Critical points and the second-derivative test #

A critical point is a point $x^*$ with $f'(x^*) = 0$. If $f$ is differentiable and has a local minimum or maximum at an inner point $x^*$, then $f'(x^*) = 0$ (Fermat's theorem: otherwise a small step in the direction $-f'(x^*)$ or $+f'(x^*)$ would go lower or higher). The converse is false.

Theorem (second-derivative test). Let $f'(x^*) = 0$ and let $f''(x^*)$ exist.

• $f''(x^*) > 0$: $x^*$ is a strict local minimum.
• $f''(x^*) < 0$: $x^*$ is a strict local maximum.
• $f''(x^*) = 0$: the test says nothing.

Proof of the first case. Taylor's theorem (next section) gives $f(x^* + h) = f(x^*) + 0 \cdot h + \tfrac12 f''(x^*) h^2 + o(h^2)$. For small $h \ne 0$ the term $\tfrac12 f''(x^*)h^2 > 0$ is larger than the $o(h^2)$ error, so $f(x^* + h) > f(x^*)$. $\blacksquare$

The third case really is undecided. All three functions $x^3$, $x^4$ and $-x^4$ have $f'(0) = f''(0) = 0$. But $x^4$ has a minimum at 0, $-x^4$ a maximum, and $x^3$ neither (it goes up through 0: an inflection point).

Taylor polynomials #

The tangent line matches $f$ and $f'$ at $x_0$. A parabola can also match $f''$. In general, the Taylor polynomial of order $k$ at $x_0$ matches the first $k$ derivatives:

$T_1$ is the tangent line. $T_2$ is the parabola that best fits the graph near $x_0$.

Theorem (Taylor, Lagrange remainder). If $f$ is $k + 1$ times differentiable on an interval that contains $x_0$ and $x_0 + h$, then

for some $\xi$ between $x_0$ and $x_0 + h$.

So if $|f^{(k+1)}| \le M$ near $x_0$, the error is at most $M |h|^{k+1}/(k+1)!$. The error has order $h^{k+1}$: halve $h$, and the error becomes about $2^{k+1}$ times smaller.

Two examples at $x_0 = 0$, with $h = 0.1$:

• $e^h \approx 1 + h + h^2/2 = 1.105$. The true value is $1.1051709\ldots$, so the error is $1.71 \cdot 10^{-4}$. The remainder is $e^\xi h^3/6$ with $0 < \xi < 0.1$: between $1.67 \cdot 10^{-4}$ and $1.84 \cdot 10^{-4}$. ✓
• $\log(1 + h) \approx h - h^2/2 = 0.095$. The true value is $0.0953102\ldots$, so the error is $3.10 \cdot 10^{-4}$, close to the next term $h^3/3 = 3.33 \cdot 10^{-4}$.

Interactive figure: loading…

Proof of Taylor's theorem with the Lagrange remainder

Write $b = x_0 + h$. Choose the number $M$ so that the function

$$g(t) = f(b) - \sum_{j=0}^{k} \frac{f^{(j)}(t)}{j!}(b - t)^j - M\,\frac{(b - t)^{k+1}}{(k+1)!}$$

has $g(x_0) = 0$. (This is one linear equation for $M$.)

1. At $t = b$ every power of $(b - t)$ is 0 except the $j = 0$ term, so $g(b) = f(b) - f(b) = 0$.
2. By Rolle's theorem there is a $\xi$ strictly between $x_0$ and $b$ with $g'(\xi) = 0$.
3. Differentiate the sum term by term with the product rule. The terms cancel in pairs (a telescoping sum), and only $-\dfrac{f^{(k+1)}(t)}{k!}(b - t)^k$ is left. So $g'(t) = -\dfrac{f^{(k+1)}(t)}{k!}(b - t)^k + M\,\dfrac{(b - t)^k}{k!}$.
4. $g'(\xi) = 0$ and $b - \xi \ne 0$ give $M = f^{(k+1)}(\xi)$.
5. Put this $M$ in $g(x_0) = 0$: this is the theorem. $\blacksquare$

First use: gradient descent in one variable #

To minimize $f$, gradient descent repeats

with a step size (learning rate) $\eta > 0$.

The sign logic. If $f'(x_k) > 0$, $f$ goes up to the right, so we step to the left. If $f'(x_k) < 0$, we step to the right. In both cases the linear approximation says $f(x_{k+1}) \approx f(x_k) - \eta f'(x_k)^2 < f(x_k)$: a small enough step always goes down.

How large may $\eta$ be? Take $f(x) = \tfrac a2 x^2$ with $a > 0$ (minimum at 0, $f'' = a$).

1. $f'(x) = a x$, so one step gives $x_{k+1} = x_k - \eta a x_k = (1 - \eta a)\, x_k$.
2. After $k$ steps: $x_k = (1 - \eta a)^k x_0$.
3. This goes to 0 for every start $x_0$ exactly when $|1 - \eta a| < 1$, that is, $\boxed{0 < \eta < 2/a}$.

The five cases:

step size	factor $1 - \eta a$	what happens
$0 < \eta < 1/a$	between 0 and 1	monotone convergence
$\eta = 1/a$	0	the minimum in one step
$1/a < \eta < 2/a$	between $-1$ and 0	jumps over 0, still converges
$\eta = 2/a$	$-1$	bounces between $x_0$ and $-x_0$ forever
$\eta > 2/a$	below $-1$	diverges

Curvature limits the step size. For a general $f$, near a minimum $x^*$ with $f''(x^*) > 0$, the Taylor polynomial says $f$ looks like a parabola with $a = f''(x^*)$. So gradient descent can settle at $x^*$ only if $\eta < 2/f''(x^*)$. Sharper minima need smaller steps.

Interactive figure: loading…

Answers to section 1. (1) $1/4$. $\sigma' = \sigma(1-\sigma)$ is largest at $\sigma = 1/2$, that is, at $x = 0$. (2) No. Here $a = 10$, so the condition is $\eta < 2/10 = 0.2$. With $\eta = 0.21$ each step multiplies $x$ by $1 - 2.1 = -1.1$: $x$ changes sign and grows, and the iterates diverge. (3) No. Near 0, ReLU is $\max(0, x)$ at every zoom level: the corner never straightens, and $r(h)/h$ does not go to 0. So ReLU has no derivative at 0.

4. Worked example #

The logistic loss of one example with label $+1$, as a function of one weight $w$ (input 1):

(a) First derivative, by the chain rule. Outer function $\log u$, inner function $u = 1 + e^{-w}$.

(We divided the top and the bottom by $e^{-w}$, then used $\sigma(-w) = 1 - \sigma(w)$.) At $w = 0$: $\ell'(0) = 1/2 - 1 = -1/2$. The derivative is negative for every $w$: the loss always decreases when $w$ grows.

(b) Second derivative. $\ell''(w) = \sigma'(w) = \sigma(w)\big(1 - \sigma(w)\big) > 0$. So $\ell$ is convex. At 0: $\ell''(0) = 1/4$.

(c) Taylor polynomial of order 2 at 0. $\ell(0) = \log 2 \approx 0.6931$, so

At $h = 1$: $T_2(1) = 0.6931 - 0.5 + 0.125 = 0.3181$. The true value is $\ell(1) = \log(1 + e^{-1}) = 0.3133$. The error is $-0.0049$. The Lagrange bound uses $|\ell'''| = |\sigma''| \le 1/(6\sqrt3) \approx 0.096$: the error is at most $0.096/6 = 0.016$. ✓

(d) One gradient-descent step from $w_0 = 0$ with $\eta = 1$: $w_1 = 0 - 1 \cdot (-1/2) = 0.5$. The loss goes from $0.6931$ to $\ell(0.5) = 0.4741$. The linear approximation predicted $0.6931 - 0.5 \cdot 0.5 = 0.4431$; the curvature makes the real decrease a bit smaller. The quadratic $T_2(0.5) = 0.4744$ is almost exact.

(e) A look ahead. $T_2$ has its minimum where $T_2'(h) = -1/2 + h/4 = 0$, at $h = 2$. Jumping to the minimum of the quadratic model is Newton's method (lesson 03). Here $\ell$ itself has no minimum: $\ell' < 0$ everywhere, and $\ell(w) \to 0$ only as $w \to \infty$.

5. In code #

NumPy checks the worked example. The central difference $\big(f(x+h) - f(x-h)\big)/(2h)$ is a numerical derivative (lesson 07 studies its error).

import numpy as np

def sigmoid(x):
    e = np.exp(-np.abs(x))                 # never exp of a large positive number
    return np.where(x >= 0, 1 / (1 + e), e / (1 + e))

loss = lambda w: np.log1p(np.exp(-w))      # logistic loss of one positive example
dloss = lambda w: sigmoid(w) - 1           # (a) first derivative
d2loss = lambda w: sigmoid(w) * (1 - sigmoid(w))   # (b) second derivative

# check (a) and (b) with central differences
h = 1e-5
print(dloss(0.0), (loss(h) - loss(-h)) / (2 * h))     # -0.5 -0.49999999999772443
print(d2loss(0.0), (dloss(h) - dloss(-h)) / (2 * h))  # 0.25 0.24999999999608666

# (c) second-order Taylor polynomial at 0, evaluated at w = 1
T2 = lambda h: np.log(2) - h / 2 + h**2 / 8
print(loss(1.0), T2(1.0), loss(1.0) - T2(1.0))        # 0.3132... 0.3181... -0.004885...

# (d) one gradient-descent step with eta = 1
w1 = 0.0 - 1.0 * dloss(0.0)
print(w1, loss(w1))                                   # 0.5 0.4740769841801067

# sigma' stays finite at |x| = 1000
x = np.array([-1000., 0., 1000.])
print(sigmoid(x) * (1 - sigmoid(x)))                  # [0.   0.25 0.  ]

The formula $e^{-x}/(1 + e^{-x})^2$ is the same function as $\sigma(x)(1 - \sigma(x))$, but in floating point it gives nan at $x = -1000$ ($e^{1000}$ overflows to inf, and inf/inf is nan). The stable sigmoid above avoids this.

6. Where it shows up #

• Activation functions. Sigmoid and tanh saturate: for large $|x|$ their derivatives are almost 0. Since $\sigma' \le 1/4$, a chain of 10 sigmoids multiplies the gradient by at most $4^{-10} \approx 10^{-6}$ (before the weights). This is one cause of vanishing gradients, and one reason ReLU and GELU replaced them in hidden layers.
• Logistic regression and classifiers. The derivative of the logistic loss with respect to the logit $z$ is $\sigma(z) - y$ for a label $y \in \{0, 1\}$: "prediction minus target". The worked example is the case $y = 1$. The same pattern appears for softmax with cross-entropy (lesson 06).
• Learning-rate limits. On a quadratic, $\eta < 2/a$. In deep networks the largest Hessian eigenvalue (the "sharpness") plays the role of $a$. Cohen et al. (2021) observed that full-batch gradient descent drives the sharpness up to about $2/\eta$ and then stays near this "edge of stability".
• Numerical methods in ML for science. The gradient-descent step on $\tfrac a2 x^2$ is the explicit Euler method for the ODE $\dot x = -a x$ with time step $\eta$. Its stability rule, $\eta < 2/a$, is the same: stiff physical systems (large $a$) force small time steps.
• Taylor expansions everywhere. Finite-difference gradient checks (lesson 07), Newton and Gauss-Newton methods (lessons 03 and 08), and the Laplace approximation (lesson 11) all start from a Taylor polynomial.

7. Common mistakes #

• Forgetting the inner derivative. $\dfrac{d}{dx} e^{-x^2/2}$ is not $e^{-x^2/2}$. The chain rule gives $-x\,e^{-x^2/2}$.
• $(fg)' = f'g'$. Wrong. The product rule is $f'g + fg'$. Check with $f = g = x$: $(x^2)' = 2x$, not $1$.
• "$f'(x^*) = 0$, so $x^*$ is a minimum." It may be a maximum or an inflection point ($x^3$ at 0).
• "$f''(x^*) = 0$, so $x^*$ is not a minimum." $x^4$ has a minimum at 0 with $f''(0) = 0$. The test is silent there.
• Computing $\sigma'$ with an unstable formula. $e^{-x}/(1 + e^{-x})^2$ gives nan for $x \approx -1000$. Use $\sigma(1 - \sigma)$ with a stable $\sigma$.
• "A larger learning rate is always faster." On $\tfrac a2 x^2$, the best rate is $\eta = 1/a$. Beyond $2/a$ the method diverges.
• $\dfrac{d}{dx}\log x = 1/x$ for all $x$. $\log x$ exists only for $x > 0$. For $x \ne 0$, $\dfrac{d}{dx}\log|x| = 1/x$.

8. Check yourself #

1. Use the limit definition to find the derivative of $f(x) = 1/x$ at $x = 2$.
   Answer

   $\dfrac{f(2+h) - f(2)}{h} = \dfrac{1}{h}\Big(\dfrac{1}{2+h} - \dfrac12\Big) = \dfrac{1}{h}\cdot\dfrac{2 - (2+h)}{2(2+h)} = \dfrac{-1}{2(2+h)}$. As $h \to 0$ this tends to $-1/4$. (Check with the rule $(x^{-1})' = -x^{-2} = -1/4$.)
2. Find the critical points of $f(x) = x^3 - 3x$ and classify them.
   Answer

   $f'(x) = 3x^2 - 3 = 0$ gives $x = \pm 1$. $f''(x) = 6x$. At $x = 1$: $f'' = 6 > 0$, a strict local minimum with $f(1) = -2$. At $x = -1$: $f'' = -6 < 0$, a strict local maximum with $f(-1) = 2$.
3. Write the order-2 Taylor polynomial of $\cos x$ at 0. Bound its error at $h = 0.1$.
   Answer

   $T_2(h) = 1 - h^2/2$. The $h^3$ coefficient of $\cos$ is 0, so $T_2 = T_3$, and the Lagrange remainder of order 3 applies: $|\cos(\xi)\,h^4/24| \le h^4/24 \approx 4.17 \cdot 10^{-6}$. The true error is $\cos(0.1) - 0.995 = 4.165 \cdot 10^{-6}$.
4. Gradient descent on $f(x) = 2x^2$ with $\eta = 0.3$, from $x_0 = 1$. Give $x_1$ and $x_2$. Does it converge?
   Answer

   $f'(x) = 4x$, so $x_{k+1} = (1 - 1.2)x_k = -0.2\,x_k$. Then $x_1 = -0.2$ and $x_2 = 0.04$. It converges, with jumps over 0, because $1/a = 0.25 < \eta = 0.3 < 2/a = 0.5$.
5. Show that $\tanh(x) = 2\sigma(2x) - 1$ gives $\tanh'(0) = 1$, and check it with $\tanh' = 1 - \tanh^2$.
   Answer

   Chain rule: $\tanh'(x) = 2 \cdot \sigma'(2x) \cdot 2 = 4\sigma'(2x)$. At 0: $4 \cdot 1/4 = 1$. Also $1 - \tanh^2(0) = 1 - 0 = 1$. ✓
6. Why does it not matter much in practice that ReLU has no derivative at 0?
   Answer

   The kink is one point, and a pre-activation hits exactly 0 only rarely. At that point frameworks use the value 0, which is a valid subgradient (any value in $[0, 1]$ is). Everywhere else the derivative exists and is 0 or 1.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. Define the derivative $f'(x)$ as a limit.

   Show answer

2. Write the linear approximation of $f$ near $x$, with its error condition.

   Show answer

3. State the chain rule in one variable.

   Show answer

4. Write $\sigma'(x)$ in terms of $\sigma(x)$. What is its largest value?

   Show answer

5. What is the derivative of softplus, $\log(1 + e^x)$?

   Show answer

6. What is $\tanh'(x)$?

   Show answer

7. What does the second-derivative test say at a critical point $x^*$?

   Show answer

8. Write the order-2 Taylor polynomial of $f$ at $x_0$ and the order of its error.

   Show answer

9. Gradient descent on $f(x) = \tfrac a2 x^2$: what does one step do to $x$, and when does it converge?

   Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "Derive the derivative of the sigmoid." — Chain rule on $(1 + e^{-x})^{-1}$, then split the fraction: $\sigma(1 - \sigma)$. Mention that it is at most $1/4$ (vanishing gradients).
• "ReLU is not differentiable at 0. Why can we train with it?" — The kink is a single point, frameworks use the subgradient value 0 there, and gradient methods only need a valid subgradient at such points.
• "What is the largest stable learning rate for $f(x) = \tfrac a2 x^2$?" — $\eta < 2/a$, because each step multiplies $x$ by $1 - \eta a$. In many dimensions: $\eta < 2/\lambda_{\max}$ of the Hessian.
• "Is a point with $f' = 0$ and $f'' = 0$ a minimum?" — The test cannot decide: compare $x^3$, $x^4$, $-x^4$.
• "Why is the derivative of the log-loss with respect to the logit $\sigma(z) - y$?" — Chain rule through $\log\sigma$, using $(\log\sigma)' = 1 - \sigma$.

10. Practice #

The notebook for this lesson runs in Colab. It has five short exercises, each with a check cell and a folded solution:

1. Implement a stable sigmoid, its derivative, $\tanh'$, softplus and its derivative; check them against central differences and at $x = \pm 1000$.
2. Differentiate a polynomial given by its coefficient array.
3. Evaluate a Taylor polynomial from a list of derivatives, and measure the error order on $e^x$.
4. Run gradient descent on $\tfrac a2 x^2$ and find the stability threshold $2/a$ by experiment.
5. Implement the second difference $\big(f(x+h) - 2f(x) + f(x-h)\big)/h^2$ and show that its error is $O(h^2)$.

11. Watch and read #

The paradox of the derivative | Chapter 2, Essence of calculus3Blue1Brown · Essence of calculus, chapter 2 · 17 min

The derivative as the limit of "rise over run"; compare with the secant figure.

Visualizing the chain rule and product rule | Chapter 4, Essence of calculus3Blue1Brown · Essence of calculus, chapter 4 · 16 min

Why the chain rule multiplies rates: the same picture as section 3.

Taylor series | Chapter 11, Essence of calculus3Blue1Brown · Essence of calculus, chapter 11 · 22 min

How each new term matches one more derivative; compare with the Taylor figure.

• Gilbert Strang, Calculus (MIT OpenCourseWare open textbook, free PDF): chapters 2–4 (derivatives, their applications and the chain rule) and chapter 10 (infinite series, including Taylor series).
• OpenStax, Calculus Volume 1 (free): chapter 3 (derivatives and the rules) and chapter 4 (maxima, minima, concavity, the second-derivative test).
• Deisenroth, Faisal and Ong, Mathematics for Machine Learning (free PDF): §5.1 (differentiation of univariate functions, Taylor series, the rules).
• Cohen et al., Gradient descent on neural networks typically occurs at the edge of stability (ICLR 2021): the rule $\eta < 2/a$ in real training.