The derivative as a linear map: Jacobians, Hessians and Taylor's theorem

Before this lesson: lesson 02 (partial derivatives, the gradient, $D_vf = \nabla f^\top v$). From linear algebra: a matrix as a map (lesson 01), the spectral theorem and the Rayleigh quotient (lesson 06), positive definite matrices (lesson 07). Time: about 75 minutes, plus the notebook.

---

Why this matters #

Almost every algorithm in ML uses a derivative. The gradient tells an optimizer which way to move. The Hessian tells how the gradient changes: it gives curvature, conditioning, saddle points, Newton and Gauss–Newton steps, and the Laplace approximation. The Jacobian is the object that backpropagation multiplies, layer by layer (lesson 04). One tool is behind all of these: Taylor's theorem. Near a point, a smooth function is a constant, plus a linear term, plus a quadratic term, plus an error that you can bound.

In ML for science the same objects have other names. The Jacobian of a weather model with respect to its initial state is its tangent linear model. The Taylor test is the standard check of the adjoint gradient of a PDE solver.

1. Predict first #

2. The picture #

One dimension first. For $f:\R\to\R$, the derivative $f'(x)$ is the slope of the tangent line (lesson 01):

The error $r(h)$ goes to zero faster than $h$. We write this as $r(h) = o(|h|)$, read "small-o of $h$". If you zoom in on the graph near $x$, it looks like a straight line.

Many dimensions. Now take a map $f:\R^n\to\R^m$, for example a layer of a network. Zoom in near a point $x$. A small input change $h \in \R^n$ gives an output change that is almost a matrix times $h$. That matrix is the derivative. It has one row per output and one column per input, so it is $m \times n$. It is called the Jacobian $Df(x)$.

The figure shows this for maps of the plane. A small square around $x_0$ goes to a curved shape. The linear map $Df(x_0)$ sends the same square to a parallelogram. When the square is small, the two shapes are the same.

Interactive figure: loading…

Read the readout: the largest relative error $\|r(h)\|/\|h\|$ on the square is roughly proportional to $\delta$. Make $\delta$ ten times smaller, and the relative error becomes ten times smaller. This is what "$r(h) = o(\|h\|)$" means in practice.

Scalar functions. When $m = 1$ (a loss), the linear map is $h \mapsto \nabla f(x)^\top h$, a dot product with the gradient $\nabla f(x) \in \R^n$. Its graph is the tangent plane. The gradient is orthogonal to the level set $\{f = f(x)\}$ and points uphill (lesson 2).

Second order. Zoom out a little. Now the tangent plane is not good enough; the function looks like a quadratic: a bowl, a dome or a saddle. The Hessian $\nabla^2 f(x)$, the $n \times n$ matrix of second derivatives, describes this shape. Its eigenvectors are the principal axes of the shape, and its eigenvalues are the curvatures along these axes.

Interactive figure: loading…

3. Definitions and main results #

The derivative is a linear map #

Definition. $f:\R^n\to\R^m$ is differentiable at $x$ if there is an $m\times n$ matrix $Df(x)$ with

$Df(x)$ is the Jacobian (the derivative) of $f$ at $x$. In words: the error of the linear model is small even compared with the step $\|h\|$. We write $r(h) = o(\|h\|)$.

The entries are partial derivatives. Take $h = t e_j$, where $e_j$ is the $j$-th unit vector, and divide by $t$:

The left side tends to $\partial f/\partial x_j$ (one partial derivative per output). And $Df(x)e_j$ is column $j$ of $Df(x)$ (linear algebra lesson 1). So column $j$ of the Jacobian is $\partial f/\partial x_j$, and

This also shows that the matrix is unique: every entry is fixed by a limit. Lesson 2 showed that partial derivatives can exist at a point where $f$ is not differentiable. The definition above is what "differentiable" means; continuous partial derivatives are enough to have it.

Object	For	Shape	Entries
Jacobian $Df(x)$	$f:\R^n\to\R^m$	$m\times n$ (outputs × inputs)	$\partial f_i/\partial x_j$; row $i$ is $\nabla f_i^\top$
gradient $\nabla f(x)$	$f:\R^n\to\R$	$n$ (a column)	$\partial f/\partial x_i$; $\nabla f = Df^\top$
Hessian $\nabla^2 f(x)$	$f:\R^n\to\R$	$n\times n$	$\partial^2 f/\partial x_i\partial x_j$; it is $D(\nabla f)$

Jacobians you will use in every backward pass #

A linear map. $f(x) = Wx$ with $W \in \R^{m\times n}$. Then $f(x+h) = Wx + Wh$ exactly, so $r(h) = 0$ and

An elementwise map. $f(x) = (\varphi(x_1), \dots, \varphi(x_n))$, for example $\tanh$ or the sigmoid applied to each entry. Output $i$ depends only on input $i$, so $\partial f_i/\partial x_j = 0$ for $i \ne j$, and

A layer. $f(x) = \tanh(Wx + b)$. Write $z = Wx + b$. Output $i$ is $\tanh(z_i)$ with $z_i = \sum_j W_{ij}x_j + b_i$. By the 1-D chain rule, $\partial f_i/\partial x_j = (1 - \tanh^2 z_i)\,W_{ij}$. In matrix form:

It is the Jacobian of the activation times the Jacobian of the linear map. Lesson 04 proves that this is always true: the Jacobian of a composition is the product of the Jacobians.

Polar coordinates. $f(r, \theta) = (r\cos\theta,\ r\sin\theta)$. The columns are the partial derivatives with respect to $r$ and $\theta$:

A small rectangle $dr \times d\theta$ goes to a region of area about $r\,dr\,d\theta$ (linear algebra lesson 3: $|\det|$ is the area scale). Lesson 10 uses this for integrals.

The Hessian and when it is symmetric #

For a scalar $f$, the Hessian is the Jacobian of the gradient: $\nabla^2 f(x) = D(\nabla f)(x)$, with entries $[\nabla^2 f]_{ij} = \partial^2 f/\partial x_i\,\partial x_j$. We write $f \in C^k$ when all partial derivatives up to order $k$ exist and are continuous.

Theorem (Schwarz, also called Clairaut). If $f$ is $C^2$ near $x$, then $\dfrac{\partial^2 f}{\partial x_i \partial x_j}(x) = \dfrac{\partial^2 f}{\partial x_j \partial x_i}(x)$. So $\nabla^2 f(x)$ is symmetric.

Why you care: a symmetric matrix has real eigenvalues and an orthonormal basis of eigenvectors (spectral theorem). Everything below uses this: curvature along eigenvectors, the minimum/maximum/saddle test, the quadratic model.

Example: a quadratic. $f(x) = \tfrac12 x^\top A x - b^\top x$ with any square $A$. Expand:

The linear term is $\big(\tfrac12(A + A^\top)x - b\big)^\top h$, because $x^\top A h = (A^\top x)^\top h$. So

Only the symmetric part of $A$ matters. "$\nabla(\tfrac12 x^\top Ax) = Ax$" is true only when $A$ is symmetric.

When the mixed partials differ

Without continuity of the second derivatives, the order can matter. Take $f(x,y) = xy(x^2-y^2)/(x^2+y^2)$ with $f(0,0) = 0$. Then $\partial_x f(0, y) = -y$ for all $y$, so $\partial_y \partial_x f(0,0) = -1$. And $\partial_y f(x, 0) = x$, so $\partial_x \partial_y f(0,0) = +1$. The second partial derivatives exist but are not continuous at $0$.

In ML: smooth losses are $C^2$, and autodiff Hessians are symmetric up to rounding. At the kinks of a ReLU network the Hessian does not exist.

Taylor's theorem #

In one dimension. If $g$ is $C^2$ near $0$, then for each $t$ there is a $\tau$ between $0$ and $t$ with

So $g(t) = g(0) + g'(0)t + \tfrac12 g''(0)t^2 + o(t^2)$. If $g$ is $C^3$, the error is $O(t^3)$: it equals $\tfrac16 g'''(\xi)\,t^3$ for some $\xi$ between $0$ and $t$ (the Lagrange form of the remainder). Here $O(t^3)$, "big-O", means "at most a constant times $|t|^3$ for small $t$".

In $n$ dimensions: reduce to a line. Fix a point $x$ and a direction $h$, and look at $f$ along the line through $x$ in direction $h$:

1. The derivative of $g$ is a directional derivative of $f$: $g'(t) = D_h f(x+th) = \nabla f(x+th)^\top h$ (lesson 2).
2. Apply the same rule to each entry of $\nabla f$: $g''(t) = h^\top \nabla^2 f(x+th)\, h$.
3. At $t = 0$: $g(0) = f(x)$, $g'(0) = \nabla f(x)^\top h$, $g''(0) = h^\top \nabla^2 f(x)\, h$.

Theorem (Taylor, second order). If $f$ is $C^2$ near $x$, then

If $f$ is $C^3$, the error is $O(\|h\|^3)$.

Proof of the error term

1. Apply the 1-D result to $g(t) = f(x+th)$ at $t = 1$: there is a $\tau \in (0, 1)$ with $f(x+h) = f(x) + \nabla f(x)^\top h + \tfrac12 h^\top \nabla^2 f(x+\tau h)\, h$.
2. Subtract the model with $\nabla^2 f(x)$. The error is $\tfrac12 h^\top \big(\nabla^2 f(x+\tau h) - \nabla^2 f(x)\big) h$.
3. Its size is at most $\tfrac12 \|\nabla^2 f(x+\tau h) - \nabla^2 f(x)\|_2\, \|h\|^2$.
4. $\nabla^2 f$ is continuous, so the matrix norm tends to $0$ as $h \to 0$. So the error divided by $\|h\|^2$ tends to 0: it is $o(\|h\|^2)$. $\blacksquare$

The three terms have the sizes $1$, $\|h\|$ and $\|h\|^2$. The Taylor test at the end of this section measures exactly these orders.

The quadratic model and the Newton step #

Write $g = \nabla f(x)$ and $H = \nabla^2 f(x)$. The quadratic model of $f$ at $x$ is

Minimize the model. Its gradient is $\nabla m(h) = g + Hh$ (the example above, with $A = H$ symmetric). If $H \succ 0$ (positive definite), $m$ is a strictly convex bowl, and its minimum is where $\nabla m = 0$:

This is the Newton step. On a quadratic $f$ with $H \succ 0$, the model is exact, so one Newton step lands on the minimum.

Gradient descent is a cruder model. Replace $H$ by $\tfrac1\eta I$. The minimizer of $f(x) + g^\top h + \tfrac{1}{2\eta}\|h\|^2$ is $h = -\eta g$: the gradient step. It uses the same curvature $1/\eta$ in every direction. In a stretched bowl this is wrong in most directions, so gradient descent zig-zags.

When $H$ is not positive definite. If $H$ is indefinite, the model is a saddle and has no minimum. Then $h_N$ is the saddle point of the model, and it can point uphill. Practical methods add damping ($H + \lambda I$ with $\lambda > 0$) or a line search.

Interactive figure: loading…

Second-order conditions for a minimum #

A critical point (stationary point) is a point $x^*$ with $\nabla f(x^*) = 0$.

Theorem (second-order conditions). Let $f$ be $C^2$ near $x^*$, with $g = \nabla f(x^*)$ and $H = \nabla^2 f(x^*)$.

• (a) Necessary. If $x^*$ is a local minimum, then $g = 0$ and $H \succeq 0$.
• (b) Sufficient. If $g = 0$ and $H \succ 0$, then $x^*$ is a strict local minimum.

Why this is true (intuition). Near $x^*$, $f$ is its quadratic model plus a small error. If the gradient were not 0, a small step against it would go down. If some direction had negative curvature, a small step along it would go down. If every direction has positive curvature, the model is a bowl, and the small error cannot cancel a bowl near its bottom.

Proof.

1. Taylor: $f(x^*+h) = f(x^*) + g^\top h + \tfrac12 h^\top H h + r(h)$ with $r(h)/\|h\|^2 \to 0$.
2. (a), gradient. Suppose $g \ne 0$. Take $h = -tg$ with $t > 0$. Then $f(x^*+h) = f(x^*) - t\|g\|^2 + O(t^2)$, which is below $f(x^*)$ for small $t$. This contradicts the local minimum, so $g = 0$.
3. (a), Hessian. Suppose $H$ has an eigenvalue $\lambda < 0$ with unit eigenvector $v$. Since $g = 0$, $f(x^*+tv) = f(x^*) + \tfrac12\lambda t^2 + o(t^2)$, which is $< f(x^*)$ for small $t \ne 0$. Contradiction. So all eigenvalues are $\ge 0$: $H \succeq 0$.
4. (b). Let $\mu = \lambda_{\min}(H) > 0$. By the Rayleigh quotient, $h^\top H h \ge \mu\|h\|^2$ for every $h$. Because $r(h)/\|h\|^2 \to 0$, there is a $\delta > 0$ with $|r(h)| \le \tfrac{\mu}{4}\|h\|^2$ for $\|h\| < \delta$.
5. Then for $0 < \|h\| < \delta$: $f(x^*+h) - f(x^*) \ge \tfrac{\mu}{2}\|h\|^2 - \tfrac{\mu}{4}\|h\|^2 > 0$. So $x^*$ is a strict local minimum. $\blacksquare$

The gap. $g = 0$ and $H \succeq 0$ is not sufficient. $f(x,y) = x^2 + y^3$ at $0$ has $\nabla^2 f = \operatorname{diag}(2, 0)$, but $f(0, -t) = -t^3 < 0$ for $t > 0$. Step 3 only rules out negative curvature. Along a direction with zero curvature, the higher-order terms decide.

Classifying a critical point ($\nabla f(x^*) = 0$, $f \in C^2$) from the eigenvalues of $H$:

Eigenvalues of $H$	Type
all $> 0$	strict local minimum
all $< 0$	strict local maximum
some $> 0$ and some $< 0$	saddle point: $f$ goes up along one eigenvector and down along another
all $\ge 0$ (or all $\le 0$), some $= 0$	degenerate: the test cannot decide

In 2-D there is a shortcut: $\det H = \lambda_1\lambda_2$. So $\det H < 0$ means a saddle, and $\det H > 0$ means a minimum ($\operatorname{tr} H > 0$) or a maximum ($\operatorname{tr} H < 0$).

Interactive figure: loading…

Answers to section 1. (1) No. $\nabla f(1,1) = (2, 20)$, so $-\nabla f = (-2, -20)$ points almost straight down along $-y$. The minimum lies in the direction $(-1,-1)$. The angle between the two is $\arccos\big(22/(\sqrt{404}\sqrt2)\big) \approx 39.3°$. This is why gradient descent zig-zags in stretched bowls. The Newton step is $-H^{-1}\nabla f = -(2/2,\ 20/20) = (-1,-1)$: it lands exactly on the minimum, because a quadratic is its own quadratic model. (2) The test cannot decide (second quiz above).

The Taylor test: check a gradient with remainder orders #

You have $f$ and code for a claimed gradient $\tilde g(x)$. You want to know if $\tilde g = \nabla f$. Pick a random direction $d$ and steps $t = 10^{-1}, 10^{-2}, \dots$. Compute

• By Taylor, $r_0(t) = O(t)$: order 1.
• If $\tilde g = \nabla f$, then $r_1(t) = O(t^2)$: order 2. Each 10× smaller $t$ makes $r_1$ 100× smaller.
• If $\tilde g$ is wrong, then $r_1(t) \approx t\,|(\nabla f - \tilde g)^\top d|$: order 1 only.
• With a claimed Hessian $\tilde H$ (or a Hessian-vector product), also subtract $\tfrac12 t^2 d^\top \tilde H d$. The remainder $r_2(t)$ has order 3 when both are correct ($f \in C^3$).

The observed order between two steps is

the slope of the line on a log–log plot.

Interactive figure: loading…

Three rules.

1. Use float64. In float32 the rounding floor is so high that the slope-2 range is short.
2. Read the order in the middle range of $t$. At very small $t$, the rounding error of $f$ (about $\varepsilon|f|$, with $\varepsilon \approx 2.2\cdot10^{-16}$ in float64) stops the decrease. That region says nothing about the gradient.
3. Use a random $d$, so that a bug is not hidden by $(\nabla f - \tilde g)^\top d = 0$.

The test needs no tolerance and no "right" step size: the order is 2 or it is not. Lesson 07 covers finite-difference checks and the best step size.

4. Worked example #

Derivatives.

The Hessian is symmetric, as Schwarz's theorem says (a polynomial is $C^\infty$).

Critical points. $\nabla f = 0$ gives $3x^2 = 3y$ and $3y^2 = 3x$, so $y = x^2$ and $x = y^2 = x^4$. Then $x(x^3 - 1) = 0$. The real solutions are $x = 0$ and $x = 1$, with $y = x^2$.

• $(0,0)$: $H = \begin{pmatrix} 0 & -3 \\ -3 & 0\end{pmatrix}$. The eigenvalues are $-3$ (eigenvector $(1,1)$) and $+3$ (eigenvector $(1,-1)$): a saddle. Check along the eigenvectors: $f(t,t) = 2t^3 - 3t^2 < 0$ and $f(t,-t) = 3t^2 > 0$ for small $t \ne 0$.
• $(1,1)$: $H = \begin{pmatrix} 6 & -3 \\ -3 & 6\end{pmatrix}$, with eigenvalues $6 \pm 3 = 3$ and $9$. Both positive: a strict local minimum, with $f(1,1) = 1 + 1 - 3 = -1$.

At the point $x_0 = (2, 1)$.

• $f(x_0) = 8 + 1 - 6 = 3$.
• $\nabla f(x_0) = (3\cdot4 - 3,\ 3\cdot1 - 6) = (9, -3)$.
• $H_0 = \begin{pmatrix} 12 & -3 \\ -3 & 6\end{pmatrix}$. Trace $18$, determinant $72 - 9 = 63$. The eigenvalues solve $\lambda^2 - 18\lambda + 63 = 0$: $\lambda = 9 \pm \sqrt{18} = 9 \pm 3\sqrt2 \approx 13.24$ and $4.76$. Both positive: $H_0$ is positive definite.
• Directional derivative along $u = (1,1)/\sqrt2$: $\nabla f^\top u = (9 - 3)/\sqrt2 = 3\sqrt2 \approx 4.243$.
• Steepest descent: $\|\nabla f\| = \sqrt{81 + 9} = 3\sqrt{10}$, so the direction is $-\nabla f/\|\nabla f\| = (-3, 1)/\sqrt{10}$, with rate $-3\sqrt{10} \approx -9.487$.

Newton step. Solve $H_0 h = -\nabla f(x_0)$:

Multiply the first line by 2 and add the second: $21 h_1 = -15$, so $h_1 = -5/7$. Then $6h_2 = 3 + 3h_1 = 3 - 15/7 = 6/7$, so $h_2 = 1/7$. The step is $h = (-5/7,\ 1/7)$.

• The new point is $x_0 + h = (9/7,\ 8/7) \approx (1.286,\ 1.143)$.
• There

• The model predicted $m(h) = 3 + g^\top h + \tfrac12 h^\top H_0 h$. At the Newton step $H_0 h = -g$, so $\tfrac12 h^\top H_0 h = -\tfrac12 g^\top h$, and

One step went from $f = 3$ to $-0.790$, near the minimum value $-1$. The model was too pessimistic here because $f$ is not quadratic.

Taylor test at $x_0$ along $d = (1, 1)$. Expand the cubic exactly:

The coefficients match the theory: $f(x_0) = 3$, $\nabla f^\top d = 9 - 3 = 6$, $\tfrac12 d^\top H_0 d = \tfrac12 (12 - 3 - 3 + 6) = 6$, and the cubic rest $2t^3$. Compare the correct gradient with a buggy one, $\tilde g = (9, -2)$, for which $\tilde g^\top d = 7$:

$t$	$r_0$	$r_1$, correct $g$	$r_1$, buggy $\tilde g$	$r_2$ (with $H_0$)
$10^{-1}$	$0.662$	$0.062$	$0.038$	$2\cdot10^{-3}$
$10^{-2}$	$0.0606$	$6.02\cdot10^{-4}$	$9.40\cdot10^{-3}$	$2\cdot10^{-6}$
$10^{-3}$	$6.01\cdot10^{-3}$	$6.00\cdot10^{-6}$	$9.94\cdot10^{-4}$	$2\cdot10^{-9}$
observed order (last two rows)	$1.00$	$2.00$	$0.98$	$3.00$

The buggy gradient shows at once: its order is 1, not 2. For it, $r_1 = |-t + 6t^2 + 2t^3|$: the $-t$ term is the bug. Here $r_2 = 2t^3$ exactly, because $f$ is a cubic.

5. In code #

NumPy checks the worked example. The notebook has these cells and the exercises.

import numpy as np
np.set_printoptions(precision=3)

def f(z):
    x, y = z
    return x**3 + y**3 - 3*x*y

def grad(z):
    x, y = z
    return np.array([3*x**2 - 3*y, 3*y**2 - 3*x])

def hess(z):
    x, y = z
    return np.array([[6*x, -3.], [-3., 6*y]])

# critical points: classify by the eigenvalues of the Hessian
for c in ([0., 0.], [1., 1.]):
    print(c, np.linalg.eigvalsh(hess(np.array(c))))  # [-3.  3.] saddle, then [3. 9.] minimum

x0 = np.array([2., 1.])
g, H = grad(x0), hess(x0)
print(f(x0), g, np.linalg.eigvalsh(H))      # 3.0 [ 9. -3.] [ 4.757 13.243]

h = np.linalg.solve(H, -g)                  # Newton step: solve, never invert
print(7 * h, round(f(x0 + h), 4))           # [-5.  1.] -0.7901

# Taylor test along d = (1, 1): observed order of r1 for the true and a buggy gradient
d = np.array([1., 1.])
for g_claim in (g, np.array([9., -2.])):
    r1 = [abs(f(x0 + t*d) - f(x0) - t * g_claim @ d) for t in (1e-2, 1e-3)]
    print(round(np.log10(r1[0] / r1[1]), 3))   # 2.001 (correct), then 0.976 (buggy)

# a Jacobian by central differences, checked against the hand formula (polar coordinates)
def polar(p):
    r, th = p
    return np.array([r*np.cos(th), r*np.sin(th)])

p0, eps = np.array([2., 0.5]), 1e-6
J_fd = np.column_stack([(polar(p0 + eps*e) - polar(p0 - eps*e)) / (2*eps) for e in np.eye(2)])
J = np.array([[np.cos(0.5), -2*np.sin(0.5)], [np.sin(0.5), 2*np.cos(0.5)]])
print(np.abs(J_fd - J).max() < 1e-8, round(np.linalg.det(J), 12))   # True 2.0 (det = r)

Automatic differentiation computes the same gradient and Hessian exactly up to rounding (lesson 05); finite differences, as in the last cell, are only approximations (lesson 07).

6. Where it shows up #

• Training and the learning rate. A gradient step minimizes a linear model plus a proximity term. On a quadratic with Hessian $H \succ 0$, gradient descent converges only for step sizes $\eta < 2/\lambda_{\max}(H)$: the top Hessian eigenvalue (the "sharpness") limits the learning rate (see the optimization track).
• Saddles and flat directions. The Hessians of large networks have many eigenvalues near zero and some negative ones. Degenerate critical points are normal there, so "$\nabla f = 0$ and $H \succeq 0$" does not certify a minimum.
• Second-order methods. Newton, Gauss–Newton, natural gradient and K-FAC all minimize a quadratic model with some curvature matrix in place of $H$. Hessian-vector products give $Hv$ without forming $H$ (lesson 08).
• Backpropagation. Every layer has a Jacobian: $W$ for a linear map, $\operatorname{diag}(\varphi')$ for an activation. Backprop multiplies their transposes with a vector (lesson 04, lesson 06).
• Bayesian deep learning. The Laplace approximation is the second-order Taylor expansion of the log posterior at its mode (lesson 11).
• ML for science. In 4D-Var data assimilation (weather forecasting), the Jacobian of the forecast model with respect to the initial state is the tangent linear model, and its transpose is the adjoint model. A physics-informed network $u(x)$ puts derivatives with respect to the input coordinates into the PDE residual, for example the Laplacian $\Delta u = \operatorname{tr} \nabla^2_x u$. Adjoint gradients of PDE solvers are verified with the Taylor test (for example in dolfin-adjoint).

7. Common mistakes #

• Row or column? MML's gradient is a row; here it is a column. A Jacobian is always outputs × inputs. Check shapes before you trust a formula.
• "All partial derivatives exist, so $f$ is differentiable." False (lesson 2). You need the definition, or continuous partial derivatives.
• $\nabla(x^\top A x) = 2Ax$ for every $A$. Only for symmetric $A$. In general the gradient is $(A + A^\top)x$ and the Hessian is $A + A^\top$.
• Classifying from the entries of $H$. $\begin{pmatrix} 1 & 2 \\ 2 & 1\end{pmatrix}$ has positive entries, but its eigenvalues are $3$ and $-1$: a saddle. Use the eigenvalues (in 2-D, $\det H < 0$ already means a saddle).
• Treating $H \succeq 0$ with a zero eigenvalue as a minimum. The test cannot decide.
• A Newton step with an indefinite $H$. It goes to the saddle of the model and can increase $f$. Check $H \succ 0$, or add damping ($H + \lambda I$ with $\lambda > 0$).
• Computing $H^{-1}$. Solve $Hh = -g$ instead; it is cheaper and more accurate.
• Taylor-test traps: float32 (the rounding floor hides order 2), reading the order where rounding dominates, or a direction $d$ that hides the bug.

8. Check yourself #

1. $f:\R^{10}\to\R^3$ and $\ell:\R^{10}\to\R$. Give the shapes of $Df(x)$, $\nabla \ell(x)$ and $\nabla^2\ell(x)$. In which spaces do $Df(x)\,v$ and $Df(x)^\top u$ live?
   Answer

   $Df$: $3\times10$; $\nabla\ell$: a vector in $\R^{10}$; $\nabla^2\ell$: $10\times10$. $Df(x)\,v \in \R^3$ (the output space; $v \in \R^{10}$). $Df(x)^\top u \in \R^{10}$ (the input space; $u \in \R^3$).
2. Find the Jacobian of $f(x, y) = (xy,\ x + y^2,\ \sin x)$ at $(0, 1)$.
   Answer

   Rows are the gradients of the three outputs: $(y, x)$, $(1, 2y)$, $(\cos x, 0)$. At $(0,1)$: $Df = \begin{pmatrix} 1 & 0 \\ 1 & 2 \\ 1 & 0 \end{pmatrix}$, a $3 \times 2$ matrix.
3. For $a, b \in \R^n$, find the gradient and the Hessian of $f(x) = (a^\top x)(b^\top x)$.
   Answer

   Product rule: $\nabla f = (b^\top x)\,a + (a^\top x)\,b$ and $\nabla^2 f = ab^\top + ba^\top$, which is symmetric, as it must be for a polynomial. A common slip is to write only $ab^\top$, which is not symmetric.
4. Classify the critical point $0$ of $f(x,y) = x^2 + 4xy + y^2$ and give a direction in which $f$ decreases.
   Answer

   $H = \begin{pmatrix}2&4\\4&2\end{pmatrix}$ has eigenvalues $6$ and $-2$: a saddle. Along the eigenvector $(1,-1)$ of $-2$: $f(t,-t) = t^2 - 4t^2 + t^2 = -2t^2 < 0$.
5. For $g = \nabla f(x) \ne 0$, when is the Newton direction $-H^{-1}g$ a descent direction?
   Answer

   When $H \succ 0$: then $H^{-1} \succ 0$, and the directional derivative along it is $g^\top(-H^{-1}g) < 0$. For an indefinite $H$, $g^\top H^{-1} g$ can be negative, and then the Newton direction points uphill.
6. In a Taylor test, the observed orders of $r_1$ are about 2 for $t$ from $10^{-1}$ to $10^{-6}$, then fall to about 0 and become noisy for smaller $t$. Is the gradient correct?
   Answer

   Yes. Order 2 in the middle range means $r_1 = O(t^2)$. At small $t$ the rounding error of $f$ (about $\varepsilon|f|$) is larger than the $t^2$ term, so $r_1$ stops decreasing. That region says nothing about the gradient.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. Define the derivative (Jacobian) of $f:\R^n\to\R^m$ at $x$.

   Show answer

2. What are the entries and the shape of the Jacobian $Df(x)$ for $f:\R^n\to\R^m$?

   Show answer

3. What is the Jacobian of an elementwise map $x \mapsto \varphi(x)$?

   Show answer

4. Compute $\det Df$ for polar coordinates $f(r,\theta) = (r\cos\theta, r\sin\theta)$.

   Show answer

5. When is the Hessian symmetric?

   Show answer

6. Write the second-order Taylor expansion of $f:\R^n\to\R$ at $x$.

   Show answer

7. What is the Newton step, and how do you compute it?

   Show answer

8. State the second-order necessary and sufficient conditions for a local minimum.

   Show answer

9. At a critical point the Hessian has eigenvalues $4$ and $-1$. What kind of point is it?

   Show answer

10. In the Taylor test, which orders do $r_1$ and $r_2$ have when the gradient and Hessian are correct?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "Why does the gradient point in the direction of steepest ascent?" — $D_v f = \nabla f^\top v$, then Cauchy–Schwarz (lesson 2).
• "What is the Jacobian of a linear layer followed by an activation?" — $\operatorname{diag}(\varphi'(z))\,W$ with $z = Wx + b$; outputs × inputs.
• "How do you tell a minimum from a saddle? And for a network with $10^8$ parameters?" — The signs of the Hessian eigenvalues. At scale, estimate the extreme eigenvalues with Hessian-vector products and power iteration or Lanczos (lesson 08).
• "Derive the Newton step. When does it fail?" — Minimize the quadratic model: $g + Hh = 0$. It fails when $H$ is not positive definite, or when the model is poor far from the minimum.
• "Is the Hessian always symmetric?" — Yes when $f$ is $C^2$ (Schwarz). It does not exist at the kinks of a ReLU network.
• "How do you test a hand-written gradient?" — The Taylor test (order 2), or a finite-difference check in float64 with a relative error (lesson 07).

10. Practice #

The notebook for this lesson runs in Colab. It has five exercises, each with a check cell and a folded solution:

1. jacobian_fd: a finite-difference Jacobian, compared with the hand Jacobians of polar coordinates and of $\tanh(Wx + b)$.
2. quadratic_gradient_hessian: gradient and Hessian of $\tfrac12 x^\top A x - b^\top x$ for a non-symmetric $A$.
3. classify_critical_point: minimum, maximum, saddle or degenerate, from the Hessian.
4. newton_step: Newton's method against gradient descent on the Rosenbrock function.
5. taylor_orders: the Taylor test; find which of three gradient functions has a planted bug.

11. Watch and read #

Local linearity for a multivariable functionKhan Academy (Grant Sanderson) · Multivariable calculus · 5 min

A map of the plane looks linear when you zoom in; this is the first figure of this lesson.

The Jacobian matrixKhan Academy (Grant Sanderson) · Multivariable calculus · 6 min

How the partial derivatives become the columns of the matrix of the local linear map.

The Hessian matrix | Multivariable calculus | Khan AcademyKhan Academy (Grant Sanderson) · Multivariable calculus · 6 min

Vector form of multivariable quadratic approximationKhan Academy (Grant Sanderson) · Multivariable calculus · 9 min

The second-order Taylor expansion written with the gradient and the Hessian, as in section 3.

Second partial derivative testKhan Academy (Grant Sanderson) · Multivariable calculus · 12 min

The 2-D test with $\det H$; this lesson states it with eigenvalues, which works in any dimension.

Lecture 1 Part 2: Derivatives as Linear OperatorsAlan Edelman and Steven G. Johnson · MIT 18.S096 Matrix Calculus for Machine Learning and Beyond · 48 min

The derivative as a linear operator, the view of this lesson, for vectors and matrices.

• Deisenroth, Faisal, Ong, Mathematics for Machine Learning (free PDF): §5.2 (partial derivatives and gradients), §5.3 (Jacobians), §5.7 (the Hessian), §5.8 (linearization and multivariate Taylor series). Note its row-gradient convention.
• Bright, Edelman, Johnson, Matrix Calculus (for Machine Learning and Beyond) (arXiv): §2 "Derivatives as linear operators" and §12 "Second derivatives, bilinear maps, and Hessian matrices".
• Nocedal and Wright, Numerical Optimization, 2nd ed. (Springer, 2006): §2.1 (Taylor's theorem and the second-order conditions) and §3.3–3.4 (Newton's method and Hessian modification).
• dolfin-adjoint, Verification: the Taylor remainder convergence test in practice.