Calculus for ML · 142

Glossary

The technical terms of this track, with short definitions (and the Persian term where available). Each term links to the lesson where it is introduced.

action: In mechanics, the functional $S[q] = \int (\text{kinetic} - \text{potential energy})\,dt$ . Its Euler–Lagrange equation is Newton's law (Hamilton's principle). Introduced in: Lesson 13
activation: The output $a^l = \varphi(z^l)$ of a layer after the elementwise activation function $\varphi$ (ReLU, tanh, ...). Introduced in: Lesson 6
adjoint: In reverse mode, the gradient $\bar u = \partial \ell/\partial u$ of the final scalar output with respect to an intermediate value $u$ . It has the same shape as $u$ . Introduced in: Lesson 4
antiderivative: A function $G$ with $G' = f$ . By the fundamental theorem of calculus, $\int_a^b f(x)\,dx = G(b) - G(a)$ . Introduced in: Lesson 10
automatic differentiation (AD): Computing exact derivatives of a program (up to rounding) by applying the chain rule to each primitive operation. The cost is a small constant times the cost of the program. Introduced in: Lesson 5
backpropagation: Reverse-mode AD applied to a neural network: one backward sweep gives the gradient of the loss with respect to every weight. Introduced in: Lesson 6
baseline (control variate): A constant $b$ subtracted in the score-function estimator, $(f(x) - b)\,s_\theta(x)$ . It keeps the estimator unbiased, because the score has mean zero, and can reduce its variance. Introduced in: Lesson 12
Bayesian information criterion (BIC): $\mathrm{BIC} = -2\log p(D \mid \hat\theta) + d\log n$ , a large- $n$ form of the Laplace evidence for $d$ parameters and $n$ data points. Smaller is better. Introduced in: Lesson 11
calculus of variations: The calculus of functionals: how $J[u]$ changes when the function $u$ changes, and the conditions for $u$ to minimize or make $J$ stationary. Introduced in: Lesson 13
catastrophic cancellation: The loss of correct digits when two nearly equal floating-point numbers are subtracted. Introduced in: Lesson 7
centering matrix: The matrix $\Pi = I - \frac1D \mathbf 1 \mathbf 1^\top$ . $\Pi x$ subtracts the mean of the entries of $x$ from each entry. Introduced in: Lesson 6
central difference: $D_0(h) = \frac{f(x+h) - f(x-h)}{2h}$ . Its truncation error is $O(h^2)$ (second order). Introduced in: Lesson 7
chain rule: The derivative of a composition is the product of the derivatives: $D(g \circ f)(x) = Dg(f(x))\,Df(x)$ . In one variable, rates multiply. Introduced in: Lesson 4
change-of-variables formula: For a diffeomorphism $T$ : $\int_{T(V)} f(x)\,dx = \int_V f(T(y))\,|\det DT(y)|\,dy$ . Small volumes scale by $|\det DT|$ . Introduced in: Lesson 10
checkpointing: Storing only some intermediate values in the forward run and recomputing the others in the backward run. It trades extra compute for less memory. Introduced in: Lesson 5
complex-step derivative: The approximation $f'(x) \approx \operatorname{Im} f(x + ih)/h$ for a real-analytic $f$ . It has no subtraction, so the step can be tiny and the result is accurate to machine precision. Introduced in: Lesson 7
computational graph: A directed acyclic graph with one node per intermediate value of a program and an edge $j \to i$ when $v_i$ is computed from $v_j$ . Introduced in: Lesson 4
condition number: For a quadratic bowl, the ratio $\kappa$ of the largest to the smallest curvature (Hessian eigenvalue). A large $\kappa$ means long thin contour ellipses and slow, zig-zag gradient descent. Introduced in: Lesson 2
continuously differentiable ( $C^1$ ): A function whose partial derivatives all exist and are continuous. Such a function is differentiable. Introduced in: Lesson 2
contour plot: A drawing of a function of two variables by its level sets (contour lines) in the plane, like the height lines of a map. Introduced in: Lesson 2
convex function: A function whose chords lie on or above its graph: $f(\lambda a + (1-\lambda) b) \le \lambda f(a) + (1-\lambda) f(b)$ for $\lambda \in [0, 1]$ . A twice differentiable function of one variable is convex exactly when $f'' \ge 0$ . Introduced in: Lesson 1
cotangent: The output weighting vector $u$ in a VJP $u^\top J$ . In reverse mode it is the adjoint that flows backward. Introduced in: Lesson 5
critical point: A point where the derivative is zero ( $f'(x^*) = 0$ , or $\nabla f(x^*) = 0$ in many variables). Every inner local minimum or maximum of a differentiable function is one, but not every critical point is an extremum. Introduced in: Lesson 1
deep equilibrium model (DEQ): A network whose output is a fixed point $z^* = f(z^*, x; \theta)$ of one layer. It is trained with an implicit gradient, so memory does not grow with the number of solver iterations. Introduced in: Lesson 9
definite integral: The limit $\int_a^b f(x)\,dx$ of Riemann sums as the strips get thin, the signed area under the graph of $f$ . Introduced in: Lesson 10
degenerate critical point: A critical point whose Hessian is singular (has a zero eigenvalue). When the other eigenvalues all have one sign, the second-order test cannot decide its type. Introduced in: Lesson 3
derivative: The number $f'(x) = \lim_{h \to 0} \frac{f(x+h) - f(x)}{h}$ , when this limit exists. It is the slope of the tangent line and the rate of change of $f$ at $x$ . Introduced in: Lesson 1
diffeomorphism: A one-to-one map $T$ onto its image such that $T$ and $T^{-1}$ both have continuous derivatives. Introduced in: Lesson 10
difference quotient: The slope $\frac{f(x+h) - f(x)}{h}$ of a secant line. The derivative is its limit as $h \to 0$ . Introduced in: Lesson 1
differentiable: A function is differentiable at $x$ if its derivative exists there, that is, if $f(x+h) = f(x) + f'(x)\,h + o(h)$ . In many variables: if a linear map $Df(x)$ with $f(x+h) = f(x) + Df(x)\,h + o(\|h\|)$ exists. Introduced in: Lesson 1
directional check: A cheap gradient check along one random direction $v$ : compare $\frac{f(x + hv) - f(x - hv)}{2h}$ with $g^\top v$ . Introduced in: Lesson 7
directional derivative: The rate of change of $f$ at $x$ along a vector $v$ : $D_v f(x) = \lim_{t \to 0} \frac{f(x + tv) - f(x)}{t}$ . For a differentiable $f$ it equals $\nabla f(x)^\top v$ . Introduced in: Lesson 2
Dirichlet energy: The functional $\int \big(\frac12 |\nabla u|^2 - f u\big)\,dx$ . Its Euler–Lagrange equation is the Poisson equation $-\Delta u = f$ . Introduced in: Lesson 13
dual number: A number $a + b\varepsilon$ with $\varepsilon^2 = 0$ . Then $f(a + b\varepsilon) = f(a) + f'(a)\,b\,\varepsilon$ , so arithmetic on dual numbers is forward-mode AD. Introduced in: Lesson 5
energy (Ritz) loss: A physics loss that minimizes the energy functional $J[u_\theta]$ of a network directly, as in the Deep Ritz method. Introduced in: Lesson 13
error signal: In backpropagation, $\delta^l = \partial \ell/\partial z^l$ , the gradient of the loss with respect to the pre-activation of layer $l$ . Introduced in: Lesson 6
essential boundary condition: A boundary condition that prescribes the value of $u$ (a Dirichlet condition). It must be imposed on the admissible functions. Introduced in: Lesson 13
Euler–Lagrange equation: The condition $L_u - \frac{d}{dx} L_{u'} = 0$ that a smooth minimizer (or stationary point) of $\int L(x, u, u')\,dx$ must satisfy. Introduced in: Lesson 13
expectation: $\mathbb E[f(X)] = \int f(x)\,p_X(x)\,dx$ , the average of $f(X)$ weighted by the density of $X$ . Introduced in: Lesson 10
exploding gradients: The gradient of early layers grows exponentially with depth because it is a product of many Jacobians that stretch vectors. Introduced in: Lesson 6
expression swell: The fast growth of the size of a formula, and of its symbolic derivative, when a program with repeated subexpressions is written as one expression. Introduced in: Lesson 5
fan-out: A value fans out when several later operations use it. Its adjoint is the sum of the contributions of all its uses. Introduced in: Lesson 4
finite difference: An approximation of a derivative by a difference of function values divided by a step, such as $(f(x+h) - f(x))/h$ . Introduced in: Lesson 7
first variation: The directional derivative of a functional: $\delta J[u; \eta] = \frac{d}{d\epsilon} J[u + \epsilon\eta]\big|_{\epsilon=0}$ . Introduced in: Lesson 13
Fisher information matrix: $F = \mathbb E[\nabla_\theta \log p_\theta(y)\,\nabla_\theta \log p_\theta(y)^\top]$ , with $y$ drawn from the model. For exponential-family outputs with natural parameters, it equals the Gauss-Newton matrix. Introduced in: Lesson 8
fixed point: A point $x^*$ with $x^* = f(x^*, \theta)$ . Its derivative is $dx^*/d\theta = (I - f_x)^{-1} f_\theta$ when $I - f_x$ is invertible. Introduced in: Lesson 9
forward difference: $D_+(h) = \frac{f(x+h) - f(x)}{h}$ . Its truncation error is $O(h)$ (first order). Introduced in: Lesson 7
forward mode: AD that carries a tangent next to each value and computes one JVP $Jv$ per run. Good for few inputs and many outputs. Introduced in: Lesson 5
functional: A map from a function $u$ to a number $J[u]$ , for example the length of a curve or an energy $\int_a^b L(x, u, u')\,dx$ . Introduced in: Lesson 13
functional derivative: The function $\delta J/\delta u$ with $\delta J[u; \eta] = \int \frac{\delta J}{\delta u}\,\eta\,dx$ for all admissible $\eta$ . It is the gradient of $J$ in the $L^2$ inner product. Introduced in: Lesson 13
Gauss-Newton matrix: For a loss $L(\theta) = \ell(r(\theta))$ , the matrix $G = J^\top \nabla^2 \ell(r)\,J$ with $J = Dr(\theta)$ . It drops the curvature of $r$ and is positive semidefinite when $\ell$ is convex. Introduced in: Lesson 8
Gaussian integral: $\int_{\R^d} e^{-\frac12 x^\top A x + b^\top x}\,dx = \sqrt{\frac{(2\pi)^d}{\det A}}\; e^{\frac12 b^\top A^{-1} b}$ for symmetric positive definite $A$ . In 1-D, $\int e^{-x^2}dx = \sqrt\pi$ . Introduced in: Lesson 11
Gaussian smoothing: Estimating $\nabla_\mu \mathbb E[f(\mu + \sigma\epsilon)] = \mathbb E[f(\mu + \sigma\epsilon)\,\epsilon]/\sigma$ from function values only (evolution strategies). Introduced in: Lesson 12
Gauss–Hermite quadrature: A $K$ -node rule $\int e^{-t^2} g(t)\,dt \approx \sum_k w_k\,g(t_k)$ that is exact for polynomials $g$ of degree up to $2K - 1$ . It turns Gaussian expectations into short sums. Introduced in: Lesson 11
gradient: The vector $\nabla f(x)$ of all partial derivatives of a scalar function $f$ . It points in the direction of steepest increase and is orthogonal to the level set through $x$ . Introduced in: Lesson 2
gradient check: A test of a gradient implementation: compare it with finite differences and require a small relative error. Introduced in: Lesson 7
gradient descent: The iteration $x_{t+1} = x_t - \eta\,\nabla f(x_t)$ : take a small step against the gradient to decrease $f$ . Introduced in: Lesson 1
gradient estimator: A random vector $\hat g$ computed from samples that estimates $\nabla_\theta \mathbb E_{p_\theta}[f(x)]$ . It is unbiased if $\mathbb E[\hat g]$ equals the true gradient. Introduced in: Lesson 12
gradient flow: The evolution $u_t = -\delta J/\delta u$ , the continuous-time limit of gradient descent on a functional. For the Dirichlet energy it is the heat equation. Introduced in: Lesson 13
Hessian: The $n \times n$ matrix $\nabla^2 f(x)$ of second partial derivatives $\partial^2 f/\partial x_i \partial x_j$ of a scalar function. It is symmetric when $f$ is $C^2$ , and it describes the curvature of $f$ . Introduced in: Lesson 3
Hessian-vector product (HVP): The product $Hv = \frac{d}{dt} \nabla f(x + tv)\big|_{t=0}$ . AD computes it exactly for a few times the cost of a gradient, without forming $H$ . Introduced in: Lesson 8
Hutchinson estimator: The trace estimate $\frac1S \sum_{s=1}^S z_s^\top A z_s$ with random probes $z_s$ such that $\mathbb E[zz^\top] = I$ . It needs only products $Az_s$ . Introduced in: Lesson 8
implicit differentiation: Differentiating the solution of an equation $F(x, \theta) = 0$ through the equation itself, with one linear solve at the solution, instead of through the solver's iterations. Introduced in: Lesson 9
implicit function theorem: If $F(x_0, \theta_0) = 0$ , $F$ is $C^1$ and $F_x$ is invertible there, then the solution $x^*(\theta)$ is a $C^1$ function near $\theta_0$ with $Dx^*(\theta) = -F_x^{-1} F_\theta$ . Introduced in: Lesson 9
implicit reparameterization: A pathwise gradient for distributions without a simple sampling path, from the CDF: $\nabla_\theta x = -\nabla_\theta C(x; \theta)/p_\theta(x)$ . Introduced in: Lesson 12
inflection point: A point where the graph changes from curving up to curving down, or back. Example: $x^3$ at 0. Introduced in: Lesson 1
iterated integral: An integral over several variables computed one variable at a time. Fubini's and Tonelli's theorems say when the order does not matter. Introduced in: Lesson 10
Jacobian: The $m \times n$ matrix $Df(x)$ of a map $f: \R^n \to \R^m$ , with entries $\partial f_i/\partial x_j$ . It is the derivative of the map: $f(x + h) \approx f(x) + Df(x)\,h$ . Introduced in: Lesson 3
Jacobian-vector product (JVP): The product $Jv$ of a Jacobian with an input direction $v$ . Forward mode computes JVPs without forming $J$ . Introduced in: Lesson 4
knowledge distillation: Training a student model on the softmax outputs of a teacher model, often with a temperature $T$ . Introduced in: Lesson 6
label smoothing: Training with a soft target distribution $q$ instead of a one-hot vector. For softmax cross-entropy the gradient with respect to the logits is $p - q$ . Introduced in: Lesson 6
Lagrangian: The integrand $L(x, u, p)$ of a functional $J[u] = \int_a^b L(x, u(x), u'(x))\,dx$ , where $p$ stands for $u'$ . Introduced in: Lesson 13
Laplace approximation: The Gaussian $N(\hat\theta, H^{-1})$ that approximates a posterior, with $\hat\theta$ the MAP estimate and $H$ the Hessian of the negative log joint there. Introduced in: Lesson 11
Laplace's method: Approximating $\int e^{-\Psi(\theta)}d\theta$ by replacing $\Psi$ with its second-order Taylor expansion at the minimum: $\approx e^{-\Psi(\hat\theta)} (2\pi)^{d/2} \det(H)^{-1/2}$ . Introduced in: Lesson 11
level set: The set $\{x : f(x) = c\}$ of all inputs where $f$ has the fixed value $c$ . In 2-D it is a contour line. Introduced in: Lesson 2
Levenberg-Marquardt: A damped Gauss-Newton method that solves $(G + \mu I)\delta = -\nabla L$ with $\mu > 0$ . Introduced in: Lesson 8
local derivative: The derivative $\partial v_i/\partial v_j$ of one primitive operation with respect to one of its inputs, with the other inputs fixed. It sits on the edge $j \to i$ of the graph. Introduced in: Lesson 4
log-derivative trick: The identity $\nabla_\theta p_\theta(x) = p_\theta(x)\,\nabla_\theta \log p_\theta(x)$ . It turns the gradient of an expectation into an expectation. Introduced in: Lesson 12
log-partition function: The log $A(z)$ of the normalizing constant of an exponential family with natural parameter $z$ . Its gradient is the mean and its Hessian is the covariance of the sufficient statistic. Introduced in: Lesson 6
log-sum-exp: The function $\operatorname{lse}(z) = \log \sum_k e^{z_k}$ . Its gradient is $\operatorname{softmax}(z)$ ; compute it stably as $m + \log\sum_k e^{z_k - m}$ with $m = \max_k z_k$ . Introduced in: Lesson 6
logits: The unnormalized scores $z \in \R^C$ of the output layer, one per class. Softmax turns them into probabilities. Introduced in: Lesson 6
machine epsilon: The relative spacing of floating-point numbers near 1: about $2.2 \cdot 10^{-16}$ in float64 and $1.2 \cdot 10^{-7}$ in float32. Introduced in: Lesson 7
MAP estimate: The maximum a posteriori estimate $\hat\theta = \arg\max_\theta p(\theta \mid D) = \arg\min_\theta \Psi(\theta)$ . Introduced in: Lesson 11
marginal likelihood (evidence): The probability of the data with the parameters integrated out: $p(D) = \int p(D \mid \theta)\,p(\theta)\,d\theta$ . Introduced in: Lesson 11
midpoint rule: A Riemann sum with the value of $f$ at the middle of each strip. Its error is $O(h^2)$ . Introduced in: Lesson 10
Monte Carlo integration: Estimating an integral as an average of the integrand at random points, for example $\int_a^b f \approx (b - a)\frac1n \sum_i f(u_i)$ with uniform $u_i$ . The error falls like $1/\sqrt n$ in any dimension. Introduced in: Lesson 10
natural boundary condition: A boundary condition that the minimization itself produces at a free end, $L_{u'} = 0$ there; for the Dirichlet energy it is $u' = 0$ . Introduced in: Lesson 13
negative log joint: $\Psi(\theta) = -\log p(D \mid \theta) - \log p(\theta)$ . In deep learning it is the training loss with weight decay, up to a constant. Introduced in: Lesson 11
Neumann series: The series $(I - A)^{-1} = I + A + A^2 + \cdots$ , which converges when the spectral radius of $A$ is below 1. Introduced in: Lesson 9
Newton step: The step $h = -H^{-1} g$ to the critical point of the quadratic model. It is the minimizer of the model when $H$ is positive definite. Introduced in: Lesson 3
normalizing flow: A generative model $x = T(z)$ with an invertible map $T$ and a simple base density, so that $\log p_X(x) = \log p_Z(z) - \log|\det DT(z)|$ is exact. Introduced in: Lesson 10
numerical differentiation: Approximating derivatives by finite differences of function values. It has truncation and rounding errors and needs about two evaluations per input. Introduced in: Lesson 5
Occam factor: In the Laplace evidence, the factor $p(\hat\theta)(2\pi)^{d/2}\det(H)^{-1/2}$ , roughly the posterior volume divided by the prior volume. It penalizes models that are more flexible than the data need. Introduced in: Lesson 11
one-hot vector: The vector $e_y$ with 1 in entry $y$ and 0 elsewhere. It encodes the true class $y$ . Introduced in: Lesson 6
one-sided derivative: The limit of the difference quotient when $h$ goes to 0 only from the right ( $h > 0$ ) or only from the left ( $h < 0$ ). The derivative exists when both exist and are equal. Introduced in: Lesson 1
operator overloading: Redefining $+$ , $\times$ and other operators for a new number type, so that ordinary code also computes derivatives. Introduced in: Lesson 5
partial derivative: The derivative of $f(x_1, \dots, x_n)$ with respect to one variable $x_i$ , with all other variables kept fixed. Written $\partial f/\partial x_i$ . Introduced in: Lesson 2
pathwise estimator (reparameterization trick): Write $x = g(\epsilon, \theta)$ with fixed noise $\epsilon$ and use $\nabla_\theta f(g(\epsilon, \theta))$ . It needs a differentiable $f$ and path, and usually has much lower variance. Introduced in: Lesson 12
pre-activation: The value $z^l = W^l a^{l-1} + b^l$ of a layer before the elementwise activation function. Introduced in: Lesson 6
precision matrix: The inverse $A = \Sigma^{-1}$ of a covariance matrix. It is the matrix in the quadratic exponent of a Gaussian density. Introduced in: Lesson 11
preconditioned gradient: A gradient multiplied by a positive definite matrix, $-P^{-1}\nabla f$ . It is the steepest direction in the norm defined by $P$ ; with $P$ the Hessian it is Newton's direction. Introduced in: Lesson 2
primitive operation: A small operation whose derivative is known in closed form, such as $+$ , $\times$ , $\exp$ , $\tanh$ or a matrix product. Programs are compositions of primitives. Introduced in: Lesson 4
probability density: A function $p_X \ge 0$ with $\int p_X = 1$ and $P(X \in B) = \int_B p_X(x)\,dx$ . It is probability per unit volume, so it can be larger than 1. Introduced in: Lesson 10
quadratic model: The second-order Taylor approximation $m(h) = f(x) + g^\top h + \frac12 h^\top H h$ with $g = \nabla f(x)$ and $H = \nabla^2 f(x)$ . Introduced in: Lesson 3
quadrature rule: An approximation of an integral by a weighted sum of function values at chosen nodes. Introduced in: Lesson 11
Rademacher vector: A random vector whose entries are independent and equal to $+1$ or $-1$ with probability 1/2 each. Introduced in: Lesson 8
relative error: For gradients, $\|g - g_{\text{num}}\|_2 / \max(\|g\|_2, \|g_{\text{num}}\|_2)$ . It does not depend on the scale of the gradient. Introduced in: Lesson 7
residual (PINN) loss: A physics loss that penalizes the squared residual of the differential equation at collocation points, as in physics-informed neural networks. Introduced in: Lesson 13
residual connection: A layer of the form $y = x + F(x)$ . Its Jacobian is $I + DF$ , so the identity path passes the gradient back unchanged. Introduced in: Lesson 4
reverse mode: AD that runs the program forward, records it, then propagates adjoints backward to compute one VJP $u^\top J$ per run. One backward run gives the full gradient of a scalar loss. Introduced in: Lesson 5
Richardson extrapolation: Combining approximations with steps $h$ and $h/2$ to cancel the leading error term, for example $(4D_0(h/2) - D_0(h))/3 = f'(x) + O(h^4)$ . Introduced in: Lesson 7
Riemann sum: The sum $\sum_i f(\xi_i)\,h$ of the areas of thin rectangles under a curve. Its limit, when it exists, is the integral. Introduced in: Lesson 10
Ritz method: Minimizing a functional over trial functions $u_c = \sum_k c_k \varphi_k$ (or a network $u_\theta$ ). The finite element method is the Ritz method with piecewise polynomials. Introduced in: Lesson 13
rounding error: The error from storing numbers with finite precision. In a finite difference it grows like $\varepsilon/h$ as the step $h$ shrinks. Introduced in: Lesson 7
saddle point: A critical point where $f$ goes up in some directions and down in others. Its Hessian has both positive and negative eigenvalues. Introduced in: Lesson 3
saturation: An activation function saturates where its derivative is almost 0, for example sigmoid and tanh for large $|x|$ . Gradients that pass through it become very small. Introduced in: Lesson 1
score: The gradient of the log-density with respect to the parameters, $s_\theta(x) = \nabla_\theta \log p_\theta(x)$ . Its mean under $p_\theta$ is zero. Introduced in: Lesson 12
score-function estimator (REINFORCE): The estimator $f(x)\,\nabla_\theta \log p_\theta(x)$ with $x \sim p_\theta$ . It needs only values of $f$ and works for discrete samples, but it is often noisy. Introduced in: Lesson 12
secant line: The line through two points of a graph, at $x$ and $x + h$ . Its slope is the difference quotient. Introduced in: Lesson 1
second derivative: The derivative $f''$ of the derivative $f'$ . It says how the slope changes, so it measures curvature. Introduced in: Lesson 1
second difference: $\frac{f(x+h) - 2f(x) + f(x-h)}{h^2} = f''(x) + O(h^2)$ , an approximation of the second derivative. Introduced in: Lesson 7
second variation: The second derivative $\frac{d^2}{d\epsilon^2} J[u + \epsilon\eta]$ at $\epsilon = 0$ . Like the Hessian, it decides whether a stationary point is a minimum. Introduced in: Lesson 13
softplus: The function $\log(1 + e^z)$ , a smooth version of ReLU. Its derivative is the sigmoid $\sigma(z)$ . Introduced in: Lesson 6
steepest descent direction: Among all unit vectors (in the Euclidean norm), $-\nabla f/\|\nabla f\|$ decreases $f$ fastest. In another norm, the steepest direction changes (for example, $-A^{-1}\nabla f$ for the norm $\sqrt{v^\top A v}$ ). Introduced in: Lesson 2
Stein's lemma: For $X \sim N(0, 1)$ and a differentiable $f$ with $\mathbb E|f'(X)| < \infty$ : $\mathbb E[X f(X)] = \mathbb E[f'(X)]$ . Introduced in: Lesson 10
step size (learning rate): The number $\eta > 0$ that scales each gradient-descent step. For $f(x) = \frac a2 x^2$ with $a > 0$ , gradient descent converges exactly when $\eta < 2/a$ . Introduced in: Lesson 1
strict local minimum: A point $x^*$ with $f(x) > f(x^*)$ for all $x \ne x^*$ near it. A critical point whose Hessian is positive definite is one. Introduced in: Lesson 3
subgradient: For a convex $f$ , a number $g$ with $f(y) \ge f(x) + g\,(y - x)$ for all $y$ . At a kink there are many; for ReLU at 0, every $g \in [0, 1]$ is one. Introduced in: Lesson 1
symbolic differentiation: Differentiation of a formula into a new formula, as in computer algebra systems. The result can grow very large (expression swell). Introduced in: Lesson 5
tangent: In forward mode, the rate of change $\dot v$ of a value $v$ when the input moves along a chosen direction. Also the name of the input vector $v$ in a JVP $Jv$ . Introduced in: Lesson 5
tangent line: The line $y = f(x_0) + f'(x_0)(x - x_0)$ that touches the graph at $x_0$ . It is the best linear approximation of $f$ near $x_0$ . Introduced in: Lesson 1
tangent plane: The graph of the linear approximation $f(x) + \nabla f(x)^\top h$ of a scalar function of several variables. Introduced in: Lesson 3
tape: The record of the operations and intermediate values of a forward run, which reverse mode reads backward. Introduced in: Lesson 5
Taylor polynomial: The polynomial of order $k$ that matches $f$ and its first $k$ derivatives at $x_0$ : $\sum_{j=0}^{k} \frac{f^{(j)}(x_0)}{j!}(x - x_0)^j$ . Introduced in: Lesson 1
Taylor test: A gradient check: the error $|f(x + th) - f(x) - t\,g^\top h|$ must shrink like $t^2$ when $g$ is the correct gradient, and only like $t$ when it is wrong. Introduced in: Lesson 3
Taylor's theorem: A smooth function is close to a polynomial near a point, with a known error. For a scalar $f$ : $f(x + h) = f(x) + \nabla f(x)^\top h + \frac12 h^\top \nabla^2 f(x)\,h + o(\|h\|^2)$ . Introduced in: Lesson 3
topological order: An order of the nodes of a graph in which every node comes after all of its parents. The forward pass visits nodes in this order and the backward pass in the reverse order. Introduced in: Lesson 4
trapezoid rule: An integration rule that joins the end values of each strip by a straight line. Its error is $O(h^2)$ . Introduced in: Lesson 10
truncation error: The error of an approximation formula in exact arithmetic, for example the gap between a secant slope and the tangent slope. It shrinks as the step $h$ shrinks. Introduced in: Lesson 7
unrolling: Differentiating through all $K$ iterations of a solver by backpropagation. It stores every iterate and differentiates the algorithm, not only the solution. Introduced in: Lesson 9
vanishing gradients: The gradient of early layers shrinks exponentially with depth because it is a product of many Jacobians with norm below 1. Introduced in: Lesson 6
vector-Jacobian product (VJP): The product $u^\top J$ (or $J^\top u$ ) of an output vector $u$ with a Jacobian $J$ . Reverse mode computes VJPs without forming $J$ . Introduced in: Lesson 4