Partial derivatives, the gradient and directional derivatives

Before this lesson: lesson 01 (derivative, linear approximation, gradient descent in one variable) and linear algebra lesson 01 (vectors). You need the dot product $u^\top v = \sum_i u_i v_i$, the length $\|v\| = \sqrt{v^\top v}$, and the angle $\theta$ between two vectors: $u^\top v = \|u\|\,\|v\|\cos\theta$. Time: about 60 minutes, plus the notebook.

---

Why this matters #

• Training. A network with $n$ weights has a loss $L:\R^n \to \R$. Gradient descent and all its variants (momentum, Adam) need the vector of $n$ partial derivatives $\nabla L$. Backpropagation computes it at the cost of a few forward passes (lessons 04–06).
• Conditioning. In a stretched bowl the gradient direction is poor, and gradient descent needs many small steps. Momentum, Adam, normalization layers and preconditioning exist largely to fix this (optimization track).
• Gradients with respect to inputs. Saliency maps use $\nabla_x$ of a class score to see which pixels matter. Adversarial examples move the input along the gradient of the loss.
• ML for science. In molecular dynamics, the force on the atoms is minus the gradient of the potential energy: $F = -\nabla_R E(R)$. Machine-learned interatomic potentials predict $E$ and get the forces by differentiation.

1. Predict first #

2. The picture #

Two ways to draw $f(x, y)$. The graph is the surface $z = f(x, y)$ over the $(x, y)$-plane. The contour plot (level-set plot) draws, in the plane, the curves where $f$ has a fixed value: the level sets $\{(x, y) : f(x, y) = c\}$ for a few values of $c$. A hiking map shows a mountain this way. Where the contour lines (for equally spaced $c$) are close together, the function is steep. In ML, think of $f$ as a loss and of $(x, y)$ as two weights.

Slices. Fix $y = y_0$. Then $x \mapsto f(x, y_0)$ is a function of one variable: the curve where the plane $y = y_0$ cuts the surface. Its slope at $x_0$ is the partial derivative $\partial f/\partial x$. Fixing $x = x_0$ instead gives the other slice and $\partial f/\partial y$.

Interactive figure: loading…

The gradient on the contour map. Draw the arrow $\nabla f = (\partial f/\partial x, \partial f/\partial y)$ at a point of the contour plot. Two facts are visible at once: the arrow points uphill (to higher contour values), and it crosses the contour line at a right angle. Its length is the steepness: long where the contour lines are close.

Interactive figure: loading…

3. Definitions and main results #

From now on, $f:\R^n \to \R$ and $x = (x_1, \dots, x_n)$. The pictures use $n = 2$ and the names $x, y$.

Partial derivatives #

Definition. The partial derivative of $f$ with respect to $x_i$ at $x$ is

where $e_i$ is the $i$-th standard basis vector. In words: change only $x_i$, keep all other variables fixed, and take the ordinary derivative. Other notations: $\partial_i f$, $\partial_{x_i} f$, $f_{x_i}$.

To compute it, treat the other variables as constants. Examples:

$f(x, y)$	$\partial f/\partial x$	$\partial f/\partial y$
$x^2 + 10y^2$	$2x$	$20y$
$x^2 y$	$2xy$	$x^2$
$e^{xy}$	$y\,e^{xy}$	$x\,e^{xy}$

For $x^2 y$ with respect to $x$, the factor $y$ is a constant, so the derivative is $y \cdot 2x$. With respect to $y$, the factor $x^2$ is the constant, so the derivative is $x^2$.

An ML example. Logistic regression with two weights $w = (w_1, w_2)$, one input $a = (a_1, a_2)$ and a label $y \in \{-1, +1\}$ has the loss $L(w) = \log\big(1 + e^{-y\,w^\top a}\big)$. Write $m = y\,w^\top a$ (the margin). The chain rule of lesson 01, with $\frac{d}{dm}\log(1 + e^{-m}) = -\sigma(-m)$ and $\partial m/\partial w_j = y\,a_j$, gives

For $a = (1, 2)$, $y = +1$ and $w = (0, 0)$: $m = 0$ and $\sigma(0) = 1/2$, so the partial derivatives are $-1/2$ and $-1$.

The gradient #

Definition. The gradient of $f$ at $x$ is the column vector of all partial derivatives:

It has the same shape as $x$: it is a vector in the input space, not a point on the surface. In the logistic example, $\nabla L(0, 0) = (-1/2, -1)^\top$, a vector in the plane of the weights.

(Some books, for example Mathematics for Machine Learning, write the gradient as a row vector. This track uses columns. Lesson 03 explains how the two fit together.)

Linear approximation in several variables #

In one variable, $f(x + h) = f(x) + f'(x)h + o(h)$. In $n$ variables the step $h$ is a vector, and the number $f'(x)$ becomes a dot product with the gradient.

Definition. $f$ is differentiable at $x$ if there is a vector $g \in \R^n$ with

Then $g = \nabla f(x)$: put $h = t e_i$, divide by $t$ and let $t \to 0$; the left side becomes $\partial f/\partial x_i$ and the right side becomes $g_i$. So the gradient is the only possible $g$. Lesson 03 extends this definition to functions with vector outputs.

Example. $f(x, y) = x^2 + 10y^2$ at $(1, 1)$, with $\nabla f = (2, 20)$ and the step $h = (0.1, -0.05)$:

• exact: $f(1.1, 0.95) = 1.21 + 9.025 = 10.235$;
• linear approximation: $f(1, 1) + \nabla f^\top h = 11 + 2 \cdot 0.1 + 20 \cdot (-0.05)$, that is, $11 + 0.2 - 1 = 10.2$;
• error: $0.035 = h_1^2 + 10h_2^2$, of order $\|h\|^2$.

Directional derivatives #

The partial derivatives measure the slope along the axes. Along any other direction $v \in \R^n$:

Definition. The directional derivative of $f$ at $x$ along $v$ is

For a unit vector $v$ ($\|v\| = 1$) it is the slope of $f$ when you walk in the direction $v$. The partial derivatives are the special cases $v = e_i$.

Theorem 1. If $f$ is differentiable at $x$, then $D_v f(x) = \nabla f(x)^\top v$ for every $v$.

Proof.

1. Put $h = tv$ in the definition of differentiability: $f(x + tv) - f(x) = t\,\nabla f(x)^\top v + r(tv)$.
2. Divide by $t \ne 0$: $\dfrac{f(x + tv) - f(x)}{t} = \nabla f(x)^\top v + \dfrac{r(tv)}{t}$.
3. For $v \ne 0$, $\left|\dfrac{r(tv)}{t}\right| = \dfrac{|r(tv)|}{\|tv\|}\,\|v\| \to 0$, because $tv \to 0$. (For $v = 0$ both sides are 0.) $\blacksquare$

So one vector, $\nabla f$, gives the slope in every direction, by a dot product. Note that $D_v f$ scales with $v$: $D_{2v} f = 2 D_v f$. To compare directions, use unit vectors.

Steepest ascent and steepest descent #

With $\theta$ the angle between $\nabla f(x)$ and a unit vector $v$, Theorem 1 gives

The slope is largest when $\cos\theta = 1$, zero when $\theta = 90°$, and smallest when $\cos\theta = -1$.

Interactive figure: loading…

Theorem 2 (steepest ascent). Let $f$ be differentiable at $x$ with $g = \nabla f(x) \ne 0$. Among all unit vectors $v$, the directional derivative $D_v f(x)$ is largest for $v = g/\|g\|$, with value $\|g\|$, and smallest for $v = -g/\|g\|$, with value $-\|g\|$.

Proof.

1. By Theorem 1, $D_v f(x) = g^\top v$.
2. The Cauchy–Schwarz inequality says $|g^\top v| \le \|g\|\,\|v\|$, with equality exactly when $v$ is a multiple of $g$. With $\|v\| = 1$: $-\|g\| \le g^\top v \le \|g\|$.
3. Equality on the right needs $v = c g$ with $c > 0$; the length 1 forces $v = g/\|g\|$. Then $g^\top v = \|g\|$. In the same way the left equality needs $v = -g/\|g\|$. $\blacksquare$

So $-\nabla f$ is the direction of steepest descent (measured with the usual length $\|v\|$), and gradient descent $x_{k+1} = x_k - \eta\,\nabla f(x_k)$ takes a step in this direction.

The gradient is perpendicular to the level set #

Let $x$ lie on the level set $\{f = c\}$, and let $\gamma(t)$ be a smooth curve that stays in the level set, with $\gamma(0) = x$. Its velocity $\gamma'(0)$ is a tangent direction of the level set.

Theorem 3. If $f$ is differentiable at $x$, then $\nabla f(x)^\top \gamma'(0) = 0$: the gradient is perpendicular to every tangent direction of the level set.

Proof. $f(\gamma(t)) = c$ for all $t$, so its derivative is 0. By the chain rule for a function along a curve, $\frac{d}{dt} f(\gamma(t)) = \nabla f(\gamma(t))^\top \gamma'(t)$ (lesson 04 proves the general chain rule). At $t = 0$ this gives $\nabla f(x)^\top \gamma'(0) = 0$. $\blacksquare$

In words: walking along a contour line, $f$ does not change, so the slope in that direction is 0. By $D_v f = \|\nabla f\|\cos\theta$, a zero slope with $\nabla f \ne 0$ means $\theta = 90°$. In 2-D, if $\nabla f = (a, b) \ne 0$, the tangent direction is $(-b, a)$.

Steepest descent zig-zags in a stretched bowl #

Take $f(x, y) = x^2 + 10y^2$ at the point $(1, 1)$. The minimum is at $(0, 0)$, in the direction $(-1, -1)$. But $-\nabla f(1, 1) = (-2, -20)$ points almost straight down. The angle between them is

The gradient is perpendicular to the contour ellipse, and an ellipse's normal does not point at its center. Gradient descent on this function multiplies each coordinate by its own factor per step: $x \leftarrow (1 - 2\eta)\,x$ and $y \leftarrow (1 - 20\eta)\,y$ (the one-variable rule of lesson 01, twice).

• Stability needs $|1 - 20\eta| < 1$, so $\eta < 0.1$. The steep direction $y$ limits the step size.
• With such a small $\eta$, the factor for $x$ is at least $1 - 0.2 = 0.8$: the flat direction converges slowly.
• The best fixed step, $\eta = 1/11$, makes both factors $\pm 9/11 \approx 0.818$. From $(1, 1)$ it needs 71 steps to reach $\|x\| < 10^{-6}$. On the round bowl $x^2 + y^2$ with $\eta = 0.5$, one step is enough.

For a stretch $\kappa$ (the function $x^2 + \kappa y^2$) the best factor is $(\kappa - 1)/(\kappa + 1)$. The number $\kappa$ is the condition number of the problem. Lesson 03 shows how the Hessian measures it, and how Newton's method removes the zig-zag.

Interactive figure: loading…

Partial derivatives are not enough #

In one variable, "the derivative exists" is the whole story. In several variables, all partial derivatives can exist at a point where the function is not differentiable. Take

1. On the $x$-axis, $f(x, 0) = 0$; on the $y$-axis, $f(0, y) = 0$. So $\partial f/\partial x(0, 0) = \partial f/\partial y(0, 0) = 0$.
2. If $f$ were differentiable at 0, its gradient would be $(0, 0)$, and $f(h) = o(\|h\|)$; in particular $f(h) \to 0$ as $h \to 0$.
3. But on the diagonal $y = x \ne 0$: $f(x, x) = x^2/(2x^2) = 1/2$. Points on the diagonal come arbitrarily close to 0, so $f$ is not even continuous at 0. It is not differentiable there.

On the line $y = mx$, $f = m/(1 + m^2)$: every line through 0 sees a different constant. The directional derivative along $(1, 1)/\sqrt2$ does not exist either: $f(tv) = 1/2$ for $t \ne 0$, so $\big(f(tv) - f(0)\big)/t = 1/(2t)$ has no limit. Choose xy/(x² + y²) in the first figure to see the surface.

Theorem 4 (a sufficient condition). If all partial derivatives of $f$ exist near $x$ and are continuous at $x$, then $f$ is differentiable at $x$. Such a function is called $C^1$ (continuously differentiable).

Proof for two variables

Let $h = (h_1, h_2)$ be small. Go from $(x, y)$ to $(x + h_1, y + h_2)$ in two moves, one variable at a time:

$$f(x + h_1, y + h_2) - f(x, y) = \big[f(x + h_1, y + h_2) - f(x, y + h_2)\big] + \big[f(x, y + h_2) - f(x, y)\big].$$

1. Each bracket is a change of a one-variable function. By the mean value theorem, the first is $\partial_x f(\xi, y + h_2)\, h_1$ for some $\xi$ between $x$ and $x + h_1$, and the second is $\partial_y f(x, \zeta)\, h_2$ for some $\zeta$ between $y$ and $y + h_2$.
2. Subtract $\nabla f(x, y)^\top h$. The rest is $r(h) = \big[\partial_x f(\xi, y + h_2) - \partial_x f(x, y)\big]h_1 + \big[\partial_y f(x, \zeta) - \partial_y f(x, y)\big]h_2$.
3. $|h_1|, |h_2| \le \|h\|$, so $|r(h)|/\|h\|$ is at most the sum of the two brackets in absolute value.
4. As $h \to 0$, the points $(\xi, y + h_2)$ and $(x, \zeta)$ go to $(x, y)$. The partial derivatives are continuous there, so both brackets go to 0. Hence $r(h)/\|h\| \to 0$. $\blacksquare$

For $n$ variables, change one coordinate at a time in the same way.

In ML practice, losses built from smooth pieces (linear maps, $\exp$, $\log$, sigmoid, softmax) are $C^1$, so their gradients behave as in this lesson. ReLU networks are differentiable everywhere except on a set of kinks of measure zero.

Steepest descent in another norm

"Steepest" means: the most decrease per unit of step length. Theorem 2 measures length with the Euclidean norm. Other norms give other directions.

• A quadratic norm $\|v\|_A = \sqrt{v^\top A v}$ with $A$ symmetric positive definite: substitute $u = A^{1/2}v$, so that $\|u\| = 1$ and $g^\top v = (A^{-1/2}g)^\top u$. Cauchy–Schwarz gives the minimizer $u = -A^{-1/2}g/\|A^{-1/2}g\|$, that is, $v = -A^{-1}g/\sqrt{g^\top A^{-1} g}$. The steepest direction is $-A^{-1}\nabla f$: a preconditioned gradient. For $x^2 + 10y^2$ with $A = \operatorname{diag}(2, 20)$ (the Hessian, lesson 03), at $(1, 1)$: $-A^{-1}(2, 20) = (-1, -1)$, exactly towards the minimum. This is Newton's direction.
• The $\ell_\infty$ norm ($|v_i| \le 1$ for all $i$): the most decrease comes from $v = -\operatorname{sign}(g)$, entry by entry. The fast gradient sign method (FGSM) for adversarial examples uses $+\varepsilon\operatorname{sign}(\nabla_x L)$ to increase the loss, and signSGD uses $-\operatorname{sign}$ of the gradient.
• The $\ell_1$ norm ($\sum_i |v_i| \le 1$): the best $v$ moves only the coordinate with the largest $|g_i|$. This is greedy coordinate descent.

Answers to section 1. (1) No. $-\nabla f(1, 1) = (-2, -20)$ is $39.3°$ away from the direction $(-1, -1)$ to the minimum, which is why gradient descent zig-zags here. (2) At a right angle (Theorem 3), whenever $\nabla f \ne 0$. (3) No. $xy/(x^2 + y^2)$ has both partial derivatives 0 at the origin but equals $1/2$ on the diagonal arbitrarily close to it.

4. Worked example #

Take $f(x, y) = x^2 y$ at the point $p = (1, 2)$, where $f(p) = 2$.

(a) Partial derivatives and gradient. With $y$ fixed, $\partial f/\partial x = 2xy$; with $x$ fixed, $\partial f/\partial y = x^2$. At $p$:

(b) Linear approximation. For the step $h = (0.1, -0.1)$: $f(p + h) \approx 2 + 4 \cdot 0.1 + 1 \cdot (-0.1) = 2.3$. Exact: $f(1.1, 1.9) = 1.21 \cdot 1.9 = 2.299$. The error is $-0.001$.

(c) A directional derivative. For the unit vector $v = (3, 4)/5$ (check: $\sqrt{9 + 16}/5 = 1$): $D_v f = (4 \cdot 3 + 1 \cdot 4)/5 = 16/5 = 3.2$. The angle to the gradient has $\cos\theta = 3.2/\sqrt{17} \approx 0.776$, so $\theta \approx 39.1°$.

(d) Steepest ascent and descent. Steepest ascent: $v = (4, 1)/\sqrt{17}$, slope $\sqrt{17} \approx 4.123$. Steepest descent: $v = -(4, 1)/\sqrt{17}$, slope $-4.123$. No unit direction has a slope outside $[-4.123, 4.123]$; the value $3.2$ of part (c) is inside, as it must be.

(e) The tangent of the level set. The level set through $p$ is $x^2 y = 2$, the curve $y = 2/x^2$. Its slope at $x = 1$ is $dy/dx = -4/x^3 = -4$, so its tangent direction is $(1, -4)$. Check: $(4, 1)^\top(1, -4) = 4 - 4 = 0$. The gradient is perpendicular to the level set. ✓ (The 2-D rule gives the tangent $(-b, a) = (-1, 4)$, the same line.)

5. In code #

NumPy checks the worked example. The partial derivatives are checked one coordinate at a time with central differences; lesson 07 explains how to choose the step.

import numpy as np

f = lambda p: p[0]**2 * p[1]                       # f(x, y) = x^2 y
grad = lambda p: np.array([2 * p[0] * p[1], p[0]**2])

x = np.array([1.0, 2.0])
g = grad(x)
print(g, np.linalg.norm(g))                        # [4. 1.] 4.123105625617661

# partial derivatives by central differences, one coordinate at a time
h = 1e-6
num = np.array([(f(x + h * e) - f(x - h * e)) / (2 * h) for e in np.eye(2)])
print(num)                                         # [4. 1.]  (up to about 1e-10)

# directional derivative: formula vs. the limit
v = np.array([3.0, 4.0]) / 5
t = 1e-6
print(g @ v, (f(x + t * v) - f(x)) / t)            # 3.2 3.20000167963741

# linear approximation
dx = np.array([0.1, -0.1])
print(f(x + dx), f(x) + g @ dx)                    # 2.2990000000000004 2.3

# the tangent of the level set is perpendicular to the gradient
tau = np.array([-g[1], g[0]]) / np.linalg.norm(g)
print(g @ tau, f(x + 1e-3 * tau) - f(x))           # 0.0 -3.5288e-07  (order t^2, not t)

The last line shows Theorem 3 numerically: a step of $10^{-3}$ along the tangent changes $f$ by about $10^{-7}$, not $10^{-3}$, because the first-order change $\nabla f^\top \tau$ is 0.

6. Where it shows up #

• Gradient descent and its variants. Every update in deep learning, $w \leftarrow w - \eta\,\nabla L(w)$ or a modified version of it, uses the vector of partial derivatives. The zig-zag of section 3 is the reason for momentum (which averages out the oscillation) and for per-coordinate step sizes as in Adam (optimization track).
• Loss-landscape pictures. Li et al. (2018) visualize a network's loss as a function of two variables, $f(\alpha, \beta) = L(w^* + \alpha\,\delta + \beta\,\eta)$ along two random directions $\delta, \eta$, and draw its contour plot: the same kind of picture as in this lesson.
• Saliency maps and adversarial examples. $\nabla_x$ of a class score with respect to the input pixels shows which pixels change the score fastest (Simonyan et al., 2013). FGSM (Goodfellow et al., 2015) adds $\varepsilon\operatorname{sign}(\nabla_x L)$ to an image: the steepest-ascent step in the $\ell_\infty$ norm (see the folded block in section 3).
• Forces from energies. Machine-learned interatomic potentials predict an energy $E(R)$ from the atom positions $R$ and compute the forces as $F = -\nabla_R E$. This guarantees a conservative force field (energy is conserved in simulations).
• Physics-informed losses. A neural field $u_\theta(x, t)$ enters a PDE residual through its partial derivatives $\partial u/\partial t$ and $\partial u/\partial x$, computed by automatic differentiation (lesson 05).

7. Common mistakes #

• Holding the wrong variable fixed. $\partial(x^2 y)/\partial x = 2xy$, not $2x$ (the factor $y$ stays) and not $x^2$ (that is $\partial/\partial y$).
• Drawing the gradient on the surface. $\nabla f(x)$ lives in the input space $\R^n$, under the surface. For $f:\R^2 \to \R$ it has 2 entries, not 3.
• Forgetting to normalize the direction. $D_v f = \nabla f^\top v$ is the slope per unit step only when $\|v\| = 1$. For $v = (1, 1)$ you get $\sqrt2$ times the slope.
• "$-\nabla f$ points at the minimum." Only in round bowls. In general it is perpendicular to the contour line, which can be far from the direction to the minimum ($39.3°$ in our example).
• "The gradient is tangent to the contour line." It is perpendicular to it. The tangent is the direction of zero slope.
• "All partial derivatives exist, so $f$ is differentiable." False: $xy/(x^2 + y^2)$ at 0. You need, for example, continuous partial derivatives.

8. Check yourself #

1. Compute $\nabla f$ for $f(x, y) = e^{xy} + x^3$ at $(1, 0)$.
   Answer

   $\partial_x f = y\,e^{xy} + 3x^2 = 0 + 3 = 3$, and $\partial_y f = x\,e^{xy} = 1$. So $\nabla f(1, 0) = (3, 1)^\top$.
2. For $f(x, y) = x^2 + y^2$ at $(3, 4)$: what is the largest slope over all unit directions, and in which direction?
   Answer

   $\nabla f = (6, 8)$, so the largest slope is $\|\nabla f\| = 10$, in the direction $(6, 8)/10 = (0.6, 0.8)$: straight away from the origin.
3. At a point, $\nabla f = (2, -1)$. Give a unit vector along which $f$ does not change to first order.
   Answer

   Any unit $v$ with $2v_1 - v_2 = 0$: $v = (1, 2)/\sqrt5$ (or its negative). This is the tangent direction of the level set.
4. Gradient descent on $f = x^2 + 4y^2$: for which step sizes $\eta$ does it converge, and which $\eta$ gives the fastest worst-case rate?
   Answer

   The factors are $1 - 2\eta$ and $1 - 8\eta$. Both have absolute value below 1 when $0 < \eta < 1/4$. The best $\eta$ balances $|1 - 2\eta| = |1 - 8\eta|$: $1 - 2\eta = 8\eta - 1$, so $\eta = 0.2$, with factor $0.6 = (\kappa - 1)/(\kappa + 1)$ for $\kappa = 4$.
5. Show that $f(x, y) = xy/(x^2 + y^2)$, $f(0, 0) = 0$, takes the value $m/(1 + m^2)$ on the line $y = mx$ ($x \ne 0$). What does this say about continuity at 0?
   Answer

   $f(x, mx) = m x^2/(x^2 + m^2x^2) = m/(1 + m^2)$, independent of $x$. Different lines give different values ($0$ for $m = 0$, $1/2$ for $m = 1$), all arbitrarily close to 0. So $f$ has no limit at 0 and is not continuous there.
6. For a differentiable $f$, why is $D_v f(x)$ linear in $v$? Give a test that can show a function is not differentiable at a point.
   Answer

   $D_v f = \nabla f^\top v$, a dot product with a fixed vector, so it is linear in $v$. Test: if some directional derivative does not exist, or if $D_{u+v} f \ne D_u f + D_v f$ for some $u, v$, then $f$ is not differentiable there.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. Define the partial derivative $\partial f/\partial x_i$ at $x$.

   Show answer

2. What is the gradient of $f:\R^n \to \R$, and what shape does it have?

   Show answer

3. Write the directional derivative in terms of the gradient.

   Show answer

4. Which unit direction gives the steepest descent, and what is the slope there?

   Show answer

5. Why is the gradient perpendicular to the level set?

   Show answer

6. Compute $\nabla(x^2 y)$ at $(1, 2)$.

   Show answer

7. Give a function with both partial derivatives at 0 that is not differentiable there.

   Show answer

8. What condition on the partial derivatives guarantees differentiability?

   Show answer

9. Why does gradient descent zig-zag on $x^2 + 10y^2$?

   Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "Why does the gradient point in the direction of steepest ascent?" — $D_v f = \nabla f^\top v$, then Cauchy–Schwarz: $\nabla f^\top v \le \|\nabla f\|$ for unit $v$, with equality only at $v = \nabla f/\|\nabla f\|$.
• "Why is the gradient perpendicular to the contour lines?" — Along a contour line $f$ is constant, so the directional derivative along it is 0, so $\nabla f^\top v = 0$.
• "Why is gradient descent slow on ill-conditioned problems, and what helps?" — The step size is limited by the steepest direction, so flat directions move slowly; the rate is about $(\kappa - 1)/(\kappa + 1)$ per step. Momentum, preconditioning (Adam's diagonal scaling) or Newton-type methods help.
• "If all partial derivatives exist, is the function differentiable?" — No; give $xy/(x^2 + y^2)$. Continuous partials are enough.
• "What is the gradient of $L(w) = \log(1 + e^{-y\,w^\top a})$?" — $-\sigma(-y\,w^\top a)\,y\,a$.

10. Practice #

The notebook for this lesson runs in Colab. It has five short exercises, each with a check cell and a folded solution:

1. Write the gradients of $x^2 + 10y^2$, $x^2 y$ and $e^{xy}$ by hand, and check them with central differences.
2. Compute a directional derivative two ways: with $\nabla f^\top v$ and with the limit.
3. Draw contours and a gradient field with quiver, and verify numerically that the gradient is perpendicular to the level set.
4. Run gradient descent on $x^2 + \kappa y^2$ and count the steps to $10^{-6}$ as $\kappa$ grows.
5. Show numerically that $xy/(x^2 + y^2)$ has different limits along different lines through 0.

11. Watch and read #

Partial derivatives, introductionGrant Sanderson · Khan Academy, multivariable calculus · 11 min

Partial derivatives as slopes of slices; compare with the surface figure.

Gradient and contour mapsGrant Sanderson · Khan Academy, multivariable calculus · 6 min

Why the gradient is perpendicular to contour lines and long where they are close.

Directional derivativeGrant Sanderson · Khan Academy, multivariable calculus · 7 min

The directional derivative as a dot product with the gradient.

Why the gradient is the direction of steepest ascentGrant Sanderson · Khan Academy, multivariable calculus · 11 min

The same argument as Theorem 2, told with pictures.

Lec 12: Gradient; directional derivative; tangent plane | MIT 18.02 Multivariable Calculus, Fall 07Denis Auroux · MIT 18.02 (OpenCourseWare) · 50 min

A full lecture: the gradient, its perpendicularity to level sets, and directional derivatives.

• Deisenroth, Faisal and Ong, Mathematics for Machine Learning (free PDF): §5.2 (partial differentiation and gradients). Note that the book writes the gradient as a row vector.
• OpenStax, Calculus Volume 3 (free): chapter 4 (functions of several variables, partial derivatives, directional derivatives and the gradient).
• MIT 18.02, Multivariable Calculus, Fall 2007 (OpenCourseWare): lecture notes and problem sets for partial derivatives and gradients.
• Li et al., Visualizing the loss landscape of neural nets (NeurIPS 2018): two-variable slices of real losses.
• Goodfellow, Shlens and Szegedy, Explaining and harnessing adversarial examples (ICLR 2015): the fast gradient sign method.