Calculus for ML: Quick recap

01 Derivatives in one variable

The derivative is the limit of difference quotients: $f'(x) = \lim_{h \to 0} \big(f(x+h) - f(x)\big)/h$ , the slope of the tangent line.
Equivalently, $f(x+h) = f(x) + f'(x)\,h + r(h)$ with $r(h)/h \to 0$ : the tangent line is the best linear approximation.
$|x|$ and ReLU have no derivative at 0 (the one-sided slopes differ); frameworks use the subgradient value $0$ there.
Rules: linearity, $(fg)' = f'g + fg'$ , and the chain rule $\big(f(g(x))\big)' = f'(g(x))\,g'(x)$ : rates multiply.
$\sigma' = \sigma(1 - \sigma) \le 1/4$ , $\tanh' = 1 - \tanh^2$ , $\frac{d}{dx}\log(1 + e^x) = \sigma(x)$ .
$f'' \ge 0$ on an interval means convex. At a critical point, $f'' > 0$ gives a strict local minimum, $f'' < 0$ a maximum, and $f'' = 0$ decides nothing.
Taylor: $f(x_0 + h) = \sum_{j \le k} f^{(j)}(x_0) h^j / j! + f^{(k+1)}(\xi) h^{k+1}/(k+1)!$ ; the error has order $h^{k+1}$ .
Gradient descent on $\tfrac a2 x^2$ multiplies $x$ by $1 - \eta a$ per step and converges exactly when $0 < \eta < 2/a$ .

02 Partial derivatives and the gradient

A function of two variables can be drawn as a surface $z = f(x, y)$ or as a contour plot of level sets $\{f = c\}$ ; close contour lines mean a steep function.
The partial derivative $\partial f/\partial x_i$ is the ordinary derivative in $x_i$ with all other variables fixed: the slope of a slice.
The gradient $\nabla f(x) = (\partial_1 f, \dots, \partial_n f)^\top$ is a column vector in the input space $\R^n$ .
$f$ is differentiable at $x$ if $f(x + h) = f(x) + \nabla f(x)^\top h + o(\|h\|)$ ; then $D_v f(x) = \nabla f(x)^\top v$ .
Among unit directions, $\nabla f/\|\nabla f\|$ gives the largest slope $\|\nabla f\|$ and $-\nabla f/\|\nabla f\|$ the smallest (Cauchy–Schwarz).
The gradient is perpendicular to the level set through its base point.
In a stretched bowl $x^2 + \kappa y^2$ , $-\nabla f$ does not point at the minimum, and gradient descent zig-zags; the best rate per step is $(\kappa - 1)/(\kappa + 1)$ .
Existing partial derivatives do not imply differentiability ( $xy/(x^2 + y^2)$ at 0); continuous partial derivatives ( $C^1$ ) do.

03 Jacobians, Hessians and Taylor

The derivative of $f:\R^n\to\R^m$ is the $m\times n$ Jacobian $Df(x)$ with $f(x+h) = f(x) + Df(x)h + o(\|h\|)$ ; its entries are $\partial f_i/\partial x_j$ .
Shapes: Jacobian outputs × inputs; gradient $\nabla f = Df^\top \in \R^n$ (a column here, a row in MML); Hessian $n \times n$ .
$D(Wx) = W$ ; an elementwise map has $D\varphi = \operatorname{diag}(\varphi'(x))$ ; polar coordinates have $\det Df = r$ .
If $f \in C^2$ , the Hessian is symmetric (Schwarz), and $f(x+h)$ equals $f(x) + \nabla f^\top h + \tfrac12 h^\top \nabla^2 f\, h$ up to an error $o(\|h\|^2)$ (Taylor).
Minimizing the quadratic model gives the Newton step: solve $Hh = -g$ . It is a descent direction when $H \succ 0$ ; with an indefinite $H$ it goes to the model's saddle.
At a critical point: all Hessian eigenvalues $> 0$ gives a strict local minimum, mixed signs a saddle; a zero eigenvalue with no negative one leaves the test undecided.
Taylor test: $|f(x+td) - f(x) - t\tilde g^\top d|$ has order 2 if $\tilde g$ is the gradient and order 1 if not; use float64, a random $d$ and the middle range of $t$ .

04 Chain rule and computational graphs

Chain rule: $D(g\circ f)(x) = Dg(f(x))\,Df(x)$ , shapes $(p\times m)(m\times n) = p\times n$ . The map applied first stands on the right; the outer Jacobian is evaluated at $f(x)$ .
Entry form: $\partial(g\circ f)_i/\partial x_j = \sum_k (\partial g_i/\partial u_k)(\partial f_k/\partial x_j)$ , a sum over the intermediate variables.
For a scalar loss, the gradient through a map is a vector-Jacobian product: $\bar x = Df(x)^\top\bar u$ , where $\bar u = \partial\ell/\partial u$ is the adjoint of $u = f(x)$ .
The three basic VJPs: $\bar x = W^\top\bar z$ for $z = Wx$ ; $\bar z = \varphi'(z)\odot\bar a$ for an elementwise $a = \varphi(z)$ ; $\bar W = \bar z\,x^\top$ (rank 1 for one sample).
Fan-out: when a value is used in several places, the contributions of all paths add, $\bar x = Df^\top\bar u + Dg^\top\bar w$ . Residual connections and shared weights are fan-out.
A computational graph has one node per primitive operation. The backward sweep $\bar v_j = \sum_{\text{children } i}\bar v_i\,\partial v_i/\partial v_j$ (seed $\bar y = 1$ ) gives all partial derivatives of one output; the forward sweep gives one directional derivative.
For $f = \ln x_1 + x_1x_2 - \sin x_2$ at $(2, 5)$ : $f = 11.652$ , $\partial f/\partial x_1 = 5.5$ , $\partial f/\partial x_2 = 2 - \cos 5 = 1.716$ .

05 Forward and reverse mode autodiff

Automatic differentiation applies the chain rule to the primitive operations of a program at the current numbers. It is exact up to rounding, has no step size, and handles loops and branches (it differentiates the branch that ran).
Forward mode carries (value, tangent) pairs; one pass gives a JVP $J\dot x$ , one column of $J$ for $\dot x = e_j$ . It needs no tape.
Dual numbers $a + b\varepsilon$ with $\varepsilon^2 = 0$ implement forward mode: $f(a + b\varepsilon) = f(a) + f'(a)\,b\,\varepsilon$ .
Reverse mode records a tape in the forward run, then accumulates $\bar v_j \mathrel{+}= \bar v_i\,\partial v_i/\partial v_j$ in reverse order; one pass gives a VJP $u^\top J$ , one row of $J$ .
One pass of either mode costs at most a small constant times one evaluation ( $c < 6$ , typically 2–3). The full Jacobian of $\R^n\to\R^m$ needs $n$ JVPs or $m$ VJPs.
Cheap gradient principle: the gradient of a scalar function costs a constant times the function, for any number of inputs. Forward mode wins when inputs are few and outputs are many.
Reverse mode stores the values of the forward run, so memory grows with depth and batch. Checkpointing every $k \approx \sqrt L$ layers lowers the peak to about $2\sqrt L$ activations for at most one extra forward pass.

06 Backprop: MLPs, losses, layers

Backprop for an MLP with softmax cross-entropy: $\delta^L = p - e_y$ , $\delta^l = \varphi'(z^l)\odot(W^{l+1})^\top\delta^{l+1}$ , $\partial\ell/\partial W^l = \delta^l(a^{l-1})^\top$ , $\partial\ell/\partial b^l = \delta^l$ .
For one sample, the gradient of a weight matrix is an outer product (rank at most 1); for a batch, $\partial\ell/\partial W^l = (\Delta^l)^\top A^{l-1}$ has rank at most $B$ .
With a mean loss over a batch, the factor $1/B$ enters once, in $\Delta^L = (P - Y)/B$ . The backward pass costs about two forward passes and needs the stored activations.
Least squares $\frac1{2n}\|Xw - y\|^2$ : gradient $\frac1n X^\top(Xw - y)$ , Hessian $\frac1n X^\top X \succeq 0$ ; one Newton step reaches the minimizer.
Logistic regression: gradient $\frac1n X^\top(\sigma(Xw) - y)$ , Hessian $\frac1n X^\top S X$ with $S = \operatorname{diag}(\sigma(1 - \sigma))$ ; convex, but on separable data no minimizer exists.
Softmax cross-entropy $\ell = \operatorname{lse}(z) - z_y$ has gradient $p - e_y$ and Hessian $\operatorname{diag}(p) - pp^\top \succeq 0$ . Compute $\operatorname{lse}(z) = m + \log\sum_k e^{z_k - m}$ with $m = \max_k z_k$ .
Layer norm backward: $dx = \frac1\sigma\big(\hat g - \operatorname{mean}(\hat g) - \hat x\,\operatorname{mean}(\hat g\odot\hat x)\big)$ ; its entries sum to 0, and it is orthogonal to $\hat x$ when $\varepsilon = 0$ .
The input-side error $\delta^1$ is a product of $L - 1$ transposed layer Jacobians, so it can vanish or explode exponentially with depth.

07 Finite differences and gradient checks

Forward difference $(f(x+h) - f(x))/h$ has truncation error $\tfrac h2 f''$ : first order. Central difference $(f(x+h) - f(x-h))/(2h)$ has $\tfrac{h^2}{6}f'''$ : second order.
Rounding adds an error of about $\varepsilon|f|/h$ , so the total error against $h$ is a V on a log–log plot; the best step is not the smallest.
Optimal steps in float64: about $\sqrt\varepsilon \approx 10^{-8}$ for the forward difference (about 8 correct digits) and $\varepsilon^{1/3} \approx 10^{-5}$ for the central one (about 10–11 digits); scale them by $\max(1, |x_i|)$ .
The complex step $\operatorname{Im} f(x+ih)/h$ has error $O(h^2)$ and no subtraction, so $h = 10^{-20}$ gives machine precision; the code must be analytic and complex-safe.
A finite-difference gradient costs $n$ or $2n$ evaluations of $f$ ; a directional check $\nabla f^\top v$ costs 2. Use them to check gradients, never to train.
A reliable gradient check uses float64, central differences, the relative error $\|g - g_{\text{num}}\|/\max(\|g\|, \|g_{\text{num}}\|)$ , a deterministic $f$ , a generic point and no kinks within $h$ .
For a smooth function in float64, a relative error below $10^{-7}$ is fine and above $10^{-3}$ is almost surely a bug.

08 Hessian-vector products

A Hessian-vector product is the derivative of the gradient along $v$ : $Hv = \frac{d}{dt}\nabla f(x + tv)\big|_{t=0}$ .
Forward-over-reverse computes $Hv$ as a JVP of the gradient; reverse-over-reverse as $\nabla(\nabla f^\top v)$ . Both are exact because $H$ is symmetric.
One exact HVP costs a small constant times one gradient (about 2–4), for any $n$ . The full Hessian needs $n$ HVPs.
A central difference of two gradients approximates $Hv$ with best relative error about $\varepsilon^{2/3}$ (about $4\times10^{-11}$ in float64).
$\nabla^2 \ell(r(\theta)) = J^\top \nabla^2\ell\, J + \sum_i \partial_i\ell\, \nabla^2 r_i$ . The Gauss-Newton matrix $G = J^\top \nabla^2\ell\, J$ keeps the first term.
$G \succeq 0$ when $\ell$ is convex, even where the Hessian is indefinite; $G = \nabla^2 L$ at a perfect fit or for a linear model. $Gv = J^\top(\nabla^2\ell\,(Jv))$ needs one JVP and one VJP.
Hutchinson: $\mathbb E[z^\top A z] = \operatorname{tr} A$ when $\mathbb E[zz^\top] = I$ . With Rademacher probes and symmetric $A$ , the variance is $2\sum_{i\ne j} A_{ij}^2$ ; Gaussian probes give $2\|A\|_F^2$ .

09 Implicit differentiation

Implicit function theorem: if $F(x_0, \theta_0) = 0$ , $F$ is $C^1$ and $F_x$ is invertible, then near $\theta_0$ the solution $x^*(\theta)$ is a unique $C^1$ map with $Dx^* = -F_x^{-1} F_\theta$ .
The derivative comes from differentiating $F(x^*(\theta), \theta) = 0$ with the chain rule; it does not depend on how $x^*$ was computed.
Fixed point $x^* = f(x^*, \theta)$ : $Dx^* = (I - f_x)^{-1} f_\theta$ , the sum of the Neumann series $\sum_j f_x^j f_\theta$ . Argmin of $h$ : $Dx^* = -(\nabla^2_{xx}h)^{-1}\nabla^2_{x\theta}h$ .
The gradient of a scalar loss needs one adjoint solve $F_x^\top u = \bar x$ and one VJP: $\nabla_\theta L = \partial_\theta \ell - F_\theta^\top u$ . CG (symmetric positive definite $F_x$ ) or GMRES do it matrix-free.
Unrolled backprop stores all $K$ iterates and has truncation error about $\rho^K$ ; the implicit gradient stores only $x^*$ and its error is proportional to the forward error $\|\hat x - x^*\|$ .
Ridge: $dw^*/d\lambda = -(X^\top X + \lambda I)^{-1} w^*$ , so $\|w^*\|$ decreases in $\lambda$ , though single weights can grow.
The theorem fails at folds ( $F_x$ singular), for non-unique or non-smooth solutions, and the gradient is poor when $F_x$ is ill-conditioned or the solver has not converged.

10 Integrals and change of variables

The integral $\int_a^b f\,dx$ is the limit of Riemann sums $\sum_i f(\xi_i)\,h$ . The fundamental theorem gives $\int_a^b f\,dx = G(b) - G(a)$ for any antiderivative $G$ . The midpoint and trapezoid rules have error $O(h^2)$ , the left sum $O(h)$ .
An expectation is an integral: $\mathbb E[f(X)] = \int f(x)\,p_X(x)\,dx$ . The Monte Carlo average of $N$ samples is unbiased with standard error $\sigma/\sqrt N$ , in any dimension.
Tonelli ( $f \ge 0$ ) and Fubini ( $\int|f| < \infty$ ) allow you to integrate one variable at a time, in any order.
In polar coordinates $dx\,dy = r\,dr\,d\theta$ , because $\det DT(r, \theta) = r$ .
Change of variables: $\int_U f(x)\,dx = \int_V f(T(y))\,|\det DT(y)|\,dy$ for a diffeomorphism $T: V \to U$ . A smooth map scales small volumes by $|\det DT|$ .
Density of $Y = g(X)$ : $p_Y(y) = p_X(x)/|\det Dg(x)|$ with $x = g^{-1}(y)$ , so $\log p_Y(y) = \log p_X(x) - \log|\det Dg(x)|$ . If $g$ is not one-to-one, add the terms of all branches.
For a composition of layers, the log-determinants add. A coupling layer has a triangular Jacobian with $\log|\det| = \sum_i s_i(x_A)$ , an $O(d)$ cost, and an exact inverse.
A nonlinear map moves the mode and the mean in a nonobvious way: for $X \sim N(0, 1)$ , the mode of $e^X$ is $e^{-1}$ and its mean is $e^{1/2}$ .

11 Gaussian integrals and Laplace

$\int_{\R} e^{-x^2}dx = \sqrt\pi$ : square the integral, use polar coordinates, and the factor $r$ in $r\,dr\,d\theta$ gives an elementary antiderivative. More generally $\int e^{-ax^2/2}dx = \sqrt{2\pi/a}$ .
For symmetric $A \succ 0$ : $\int_{\R^d} e^{-\frac12 x^\top A x + b^\top x}dx = (2\pi)^{d/2}\det(A)^{-1/2}e^{\frac12 b^\top A^{-1}b}$ . Proof: complete the square, shift, rotate to the eigenvectors, factor.
The normalized density is $N(A^{-1}b, A^{-1})$ : $A$ is the precision matrix. The gradient of $\log Z(b)$ is the mean and its Hessian is the covariance.
The marginal likelihood of Bayesian linear regression is a Gaussian integral over the weights; its $-\tfrac12\log\det$ term penalizes flexible models.
Gauss–Hermite: $\mathbb E[f(X)] \approx \frac{1}{\sqrt\pi}\sum_k w_k f(\mu + \sqrt2\,\sigma t_k)$ for $X \sim N(\mu, \sigma^2)$ ; $K$ nodes are exact for polynomials of degree $\le 2K - 1$ .
Laplace's method: $\int e^{-n\varphi} = e^{-n\varphi(x^*)}(2\pi/n)^{d/2}\det(\nabla^2\varphi(x^*))^{-1/2}(1 + O(1/n))$ for a unique, nondegenerate minimum. It gives Stirling's formula $n! \approx \sqrt{2\pi n}(n/e)^n$ , with relative error $\approx 1/(12n)$ .
Laplace approximation: $p(\theta \mid D) \approx N(\hat\theta_{\text{MAP}}, H^{-1})$ and $\log p(D) \approx -\Psi(\hat\theta) + \tfrac d2\log 2\pi - \tfrac12\log\det H$ , with $\Psi$ the negative log joint and $H$ its Hessian. Keeping only the terms that grow with $n$ gives the BIC.
The Laplace approximation is exact for linear-Gaussian models. It fails for multimodal, skewed or bounded posteriors, modes on a boundary or at a kink, and singular or indefinite Hessians, and it depends on the parameterization.

12 Gradients of expectations

When $\theta$ is in the distribution, $F(\theta) = \mathbb E_{p_\theta}[f(x)]$ changes because the density moves; autodiff through a plain sample returns $0$ or an error.
Score-function identity (REINFORCE): $\nabla_\theta F = \mathbb E[f(x)\,\nabla_\theta \log p_\theta(x)]$ . It needs only values of $f$ and works for discrete $x$ , but the support of $p_\theta$ must not depend on $\theta$ .
The score has mean zero, $\mathbb E[\nabla_\theta \log p_\theta(x)] = 0$ , so a baseline $b$ that does not depend on $x$ keeps the estimator unbiased; the best constant is $b^* = \mathbb E[f s^2]/\mathbb E[s^2]$ .
Pathwise gradient (reparameterization): with $x = g(\epsilon, \theta)$ , $\nabla_\theta F = \mathbb E_\epsilon[(\partial g/\partial\theta)^\top \nabla_x f(x)]$ ; for a Gaussian, $x = \mu + \sigma\epsilon$ , $\partial x/\partial\mu = 1$ , $\partial x/\partial\sigma = \epsilon$ .
The pathwise estimator needs a differentiable path and an $f$ without jumps; through a jump it is biased.
For $x \sim N(2, 1)$ , $f = x^2$ : per-sample variances $4$ (pathwise), $87$ (score), $38$ (score with $b^* = 7$ ); all means are $4$ .
Score-function variance grows with $|f|$ and with the dimension; pathwise variance follows the slope of $f$ and can be larger for fast-oscillating $f$ .
VAE encoder: pathwise. Policy gradients, discrete choices, black-box simulators: score function with a baseline.

13 Calculus of variations

A functional $J[u] = \int_a^b L(x, u, u')\,dx$ maps a function to a number; on a grid it becomes an ordinary function of the values $U_i$ , and $\partial J_h/\partial U_i \approx h\,\delta J/\delta u(x_i)$ .
The first variation is $\delta J[u; \eta]$ $= \frac{d}{d\epsilon}J[u + \epsilon\eta]\big|_{\epsilon=0}$ $= \int (L_u\eta + L_p\eta')\,dx$ , the directional derivative in the direction of a function $\eta$ .
Integration by parts and the fundamental lemma give the Euler–Lagrange equation $L_u - \frac{d}{dx}L_p = 0$ , where $\frac{d}{dx}$ is a total derivative; so $\delta J/\delta u = L_u - \frac{d}{dx}L_p$ .
At a free end the minimizer satisfies the natural boundary condition $L_p = 0$ (for $\tfrac12u'^2$ : $u' = 0$ ); prescribed values are essential conditions.
Euler–Lagrange is necessary only; if $L$ is convex in $(u, p)$ , every solution is a global minimizer.
The shortest path is a straight line; the Dirichlet energy $\int(\tfrac12|\nabla u|^2 - fu)$ gives the Poisson equation $-\Delta u = f$ .
Energy (Ritz) losses need half the derivatives and get natural conditions for free; residual (PINN) losses work for any PDE but have Hessian like $A^\top A$ , with condition number $\kappa(A)^2$ .
Gradient descent on a discretized functional is an explicit Euler step of the gradient flow $u_t = -\delta J/\delta u$ ; for the Dirichlet energy this is the heat equation, stable only for $\tau \le h/2$ .