Optimization · 125

Glossary

The technical terms of this track, with short definitions (and the Persian term where available). Each term links to the lesson where it is introduced.

(L0, L1)-smoothness: The condition $\|\nabla^2 f(x)\| \le L_0 + L_1\|\nabla f(x)\|$ : curvature may grow with the gradient norm. Under it, clipped gradient descent is much faster than gradient descent with a fixed step. Introduced in: Lesson 7
active constraint: An inequality constraint $f_i(x) \le 0$ that holds with equality at the point, $f_i(x) = 0$ . An inactive constraint has $f_i(x) < 0$ and multiplier zero. Introduced in: Lesson 8
AdaGrad: An adaptive method that divides each coordinate of the gradient by the square root of the sum of all its past squared gradients. Its step sizes only shrink. Introduced in: Lesson 6
Adam: The update $\theta_t = \theta_{t-1} - \eta\,\hat m_t/(\sqrt{\hat v_t} + \varepsilon)$ , with bias-corrected EMAs $\hat m_t$ of the gradient and $\hat v_t$ of its square. Defaults $\beta_1 = 0.9$ , $\beta_2 = 0.999$ . Introduced in: Lesson 6
AdamW: Adam with decoupled weight decay (Loshchilov and Hutter 2019). The decay $\eta\lambda\theta$ is applied to the weights directly and is not divided by $\sqrt{\hat v_t}$ . Introduced in: Lesson 6
adjoint variables: The Lagrange multipliers of the state equation in a PDE-constrained problem. One adjoint solve gives the gradient with respect to all parameters. Introduced in: Lesson 8
ADMM: The alternating direction method of multipliers for $\min f(x) + g(z)$ with $Ax + Bz = c$ . It minimizes an augmented Lagrangian over $x$ , then over $z$ , then updates the dual variable. Introduced in: Lesson 9
affine invariance: Newton's method gives corresponding iterates on $f(x)$ and on $f(Ty)$ for an invertible $T$ , so scaling or rotating the variables does not change its speed. Introduced in: Lesson 10
Armijo condition: The sufficient-decrease test $f(x + \alpha p) \le f(x) + c_1\alpha\nabla f(x)^\top p$ with a small $c_1 \in (0, 1)$ , for example $10^{-4}$ . Introduced in: Lesson 3
backtracking: A line search that starts with a trial step and multiplies it by $\tau \in (0, 1)$ until the Armijo condition holds. Introduced in: Lesson 3
BFGS: The quasi-Newton update $H^+ = (I - \rho sy^\top)H(I - \rho ys^\top) + \rho ss^\top$ with $\rho = 1/(y^\top s)$ . It keeps $H$ positive definite when $y^\top s > 0$ . Introduced in: Lesson 10
bias correction: Division of an EMA started at zero by $1 - \beta^t$ , so that early estimates are not too small. Adam applies it to both moments. Introduced in: Lesson 6
complementary slackness: The condition $\lambda_i f_i(x) = 0$ for each inequality: a constraint can have a nonzero multiplier only when it is active. Introduced in: Lesson 8
condition number: The ratio $\kappa = L/\mu$ of the largest to the smallest curvature. For a quadratic with Hessian $A$ , $\kappa = \lambda_{\max}(A)/\lambda_{\min}(A)$ . Large $\kappa$ means slow first-order methods. Introduced in: Lesson 2
conjugate gradient method: A Krylov method for $Ax = b$ with $A$ symmetric positive definite. Each step is optimal over a Krylov subspace, and the error bound shrinks like $\big((\sqrt\kappa - 1)/(\sqrt\kappa + 1)\big)^k$ . Introduced in: Lesson 4
constraint qualification: A condition on the constraints, such as Slater's condition or LICQ, that guarantees that a minimizer satisfies the KKT conditions. Introduced in: Lesson 8
convex function: A function whose chord lies above its graph, $f(\theta x + (1 - \theta)y) \le \theta f(x) + (1 - \theta)f(y)$ for $\theta \in [0, 1]$ . Every local minimum of a convex function is global. Introduced in: Lesson 2
cooldown: The final decay phase of a schedule such as WSD, where the learning rate falls to a small value, for example linearly or with the $1-\sqrt{\cdot}$ shape. Introduced in: Lesson 7
cosine schedule: The decay $\eta(t) = \eta_{\min} + \tfrac12(\eta_{\max} - \eta_{\min})(1 + \cos\pi p)$ with $p$ the fraction of the decay phase done. It needs the total length $T$ in advance. Introduced in: Lesson 7
critical batch size: The batch size above which a larger batch no longer reduces the number of steps much, so it wastes computation. It is close to the gradient noise scale. Introduced in: Lesson 5
damped Newton method: Newton's method with a line search on the Newton direction, so that $f$ decreases enough at every step. Introduced in: Lesson 10
descent direction: A direction $p$ with $\nabla f(x)^\top p < 0$ . A small enough step along $p$ decreases $f$ . Introduced in: Lesson 3
descent lemma: For an $L$ -smooth $f$ , $f(y) \le f(x) + \nabla f(x)^\top(y - x) + \tfrac L2\|y - x\|^2$ . The step $\eta = 1/L$ minimizes this upper bound. Introduced in: Lesson 2
diagonal preconditioner: A diagonal matrix $D$ that rescales each coordinate, so the method runs on $f(Dy)$ instead of $f(x)$ . Feature standardization is one; Adam uses an adaptive one. Introduced in: Lesson 2
directional derivative: The slope of $f$ along a unit vector $u$ , equal to $\nabla f(x)^\top u$ . It is largest for $u = \nabla f(x)/\|\nabla f(x)\|$ . Introduced in: Lesson 1
dual function: The function $g(\lambda, \nu) = \inf_x \mathcal L(x, \lambda, \nu)$ . It is concave, and for $\lambda \ge 0$ it is a lower bound on $f^\star$ . Introduced in: Lesson 8
dual norm: For a norm $\|\cdot\|$ , the norm $\|g\|_* = \max_{\|\Delta\| \le 1}\langle g, \Delta\rangle$ . The dual of $\ell_\infty$ is $\ell_1$ , and the dual of the spectral norm is the nuclear norm. Introduced in: Lesson 12
dual problem: The problem of finding the best lower bound, $\max_{\lambda \ge 0, \nu} g(\lambda, \nu)$ . Its optimal value $d^\star$ is never larger than $p^\star$ (weak duality). Introduced in: Lesson 8
duality gap: The difference between a primal value and a dual value, for example $p^\star - d^\star \ge 0$ . A small gap at a pair of points certifies that both are nearly optimal. Introduced in: Lesson 8
edge of stability: The regime where the sharpness reaches $2/\eta$ and stays near it. The loss oscillates over short times but keeps decreasing over long times (Cohen et al. 2021). Introduced in: Lesson 11
empirical risk: The average loss over the $n$ training examples, $f(x) = \frac1n\sum_{i=1}^n f_i(x)$ . Introduced in: Lesson 5
epoch: One full pass over the training data. Introduced in: Lesson 5
exact line search: A line search that chooses the step $\alpha$ that minimizes $f(x + \alpha p)$ exactly. On a quadratic it has a closed form. Introduced in: Lesson 3
exponential moving average (EMA): The running average $m_t = \beta m_{t-1} + (1 - \beta)g_t$ with $m_0 = 0$ . It weights the gradient of $j$ steps ago by $(1 - \beta)\beta^j$ . Introduced in: Lesson 6
feasible point: A point that satisfies all constraints of the problem. Introduced in: Lesson 8
Fisher block: The part of the Fisher matrix for the weights of one layer. For a layer with $n$ inputs and $m$ outputs it has $(mn)^2$ entries. Introduced in: Lesson 12
Fisher information matrix: The expected outer product of the gradient of the log-likelihood, with the labels sampled from the model. For losses such as cross-entropy it equals the GGN. Introduced in: Lesson 10
FISTA: The proximal gradient method with Nesterov momentum. It reaches $F(x_k) - F^\star = O(1/k^2)$ , but $F$ need not decrease at every step. Introduced in: Lesson 9
gain ratio: The ratio of the actual decrease of the function to the decrease the model predicted. Levenberg–Marquardt and trust-region methods use it to accept steps and adapt $\lambda$ or the radius. Introduced in: Lesson 10
Gauss–Newton method: For $F = \tfrac12\|r(x)\|^2$ , the step solves $J^\top J\,h = -J^\top r$ . It is Newton's method without the term $\sum_i r_i\nabla^2 r_i$ of the Hessian. Introduced in: Lesson 10
generalized Gauss–Newton (GGN) matrix: The matrix $J_f^\top\,\nabla^2\ell\,J_f$ , the part of the Hessian of $\ell(f(\theta))$ that ignores the curvature of the network $f$ . It is positive semidefinite when $\ell$ is convex. Introduced in: Lesson 10
global minimum: The smallest value of $f$ over all points, and a point where it is reached. Introduced in: Lesson 1
global norm: The Euclidean norm of all parameter gradients taken together, $\|g\| = \big(\sum_i\|g^{(i)}\|^2\big)^{1/2}$ . Introduced in: Lesson 7
gradient clipping: Limiting the size of the gradient before the step. Clipping by global norm replaces $g$ by $g\min(1, c/\|g\|)$ , which keeps its direction. Introduced in: Lesson 7
gradient descent: The method $x_{k+1} = x_k - \eta\nabla f(x_k)$ . Each step moves against the gradient, the direction of steepest decrease. Introduced in: Lesson 1
gradient noise scale: The quantity $B_{\text{noise}} = \operatorname{tr}(H\Sigma)/(G^\top HG)$ that compares gradient noise with the signal. The simple noise scale $\operatorname{tr}(\Sigma)/\|G\|^2$ approximates it. Introduced in: Lesson 5
heavy-ball method: Polyak's method $x_{k+1} = x_k - \eta\nabla f(x_k) + \beta\,(x_k - x_{k-1})$ . With the best $\eta$ and $\beta$ on a quadratic, its factor per step is $(\sqrt\kappa - 1)/(\sqrt\kappa + 1)$ . Introduced in: Lesson 4
Hessian-vector product (HVP): The product $\nabla^2 f(x)\,v$ , computed by automatic differentiation for a small multiple of the cost of one gradient, without forming the Hessian. Introduced in: Lesson 11
indicator function: The function $\iota_C(x)$ that is 0 on a set $C$ and $+\infty$ outside it. Adding it to $f$ turns a constraint $x \in C$ into a penalty. Introduced in: Lesson 9
iteration matrix: The matrix $T$ that maps the state $(x_k, x_{k-1})$ to the next state for a linear method on a quadratic. The method converges when its spectral radius is below 1. Introduced in: Lesson 4
K-FAC: Kronecker-factored approximate curvature. It approximates the Fisher block of a linear layer by $A \otimes D$ (input and output-gradient covariances) and takes the step $D^{-1}\nabla_W A^{-1}$ . Introduced in: Lesson 12
KKT conditions: The Karush–Kuhn–Tucker conditions: primal feasibility, dual feasibility $\lambda \ge 0$ , complementary slackness and stationarity of the Lagrangian in $x$ . For a convex problem with strong duality they are necessary and sufficient. Introduced in: Lesson 8
Kronecker product: For $A \in \R^{m \times n}$ and $B$ , the block matrix $A \otimes B$ with blocks $a_{ij}B$ . It satisfies $(A \otimes B)^{-1} = A^{-1} \otimes B^{-1}$ . Introduced in: Lesson 12
L-BFGS: Limited-memory BFGS. It stores only the $m$ newest pairs $(s_i, y_i)$ and computes $-H_k\nabla f$ with the two-loop recursion in $O(md)$ time. Introduced in: Lesson 10
L-smooth: A function whose gradient is Lipschitz with constant $L$ , $\|\nabla f(x) - \nabla f(y)\| \le L\|x - y\|$ . For a twice differentiable $f$ this means $\nabla^2 f \preceq LI$ everywhere (and $\succeq -LI$ ). Introduced in: Lesson 2
L2 regularization: Adding $\tfrac{\lambda}{2}\|\theta\|^2$ to the loss, so the gradient gets the extra term $\lambda\theta$ . Inside Adam this term is divided by $\sqrt{\hat v_t}$ like the rest of the gradient. Introduced in: Lesson 6
Lagrange multiplier: The coefficient $\lambda_i$ or $\nu_j$ of a constraint in the Lagrangian. At the optimum it is the price of the constraint, the rate at which $f^\star$ changes when the constraint is relaxed. Introduced in: Lesson 8
Lagrangian: The function $\mathcal L(x, \lambda, \nu) = f_0(x) + \sum_i\lambda_i f_i(x) + \sum_j\nu_j h_j(x)$ that adds the constraints to the objective with multipliers. Introduced in: Lesson 8
lasso: Least squares with an $\ell_1$ penalty, $\min_x \tfrac12\|Ax - b\|^2 + \lambda\|x\|_1$ . Its solutions are often sparse. Introduced in: Lesson 9
learning rate (step size): The number $\eta > 0$ that multiplies the gradient in a gradient-descent step. On $\tfrac{\lambda}{2}x^2$ , gradient descent is stable only for $\eta < 2/\lambda$ . Introduced in: Lesson 1
learning-rate range test: A short run where the learning rate grows (often exponentially) while a smoothed loss is recorded. The peak learning rate is chosen clearly below the point where the loss blows up. Introduced in: Lesson 7
learning-rate schedule: A function $\eta(t)$ that gives the learning rate at each step $t$ of training. Introduced in: Lesson 7
level set: The set of points where $f$ has one fixed value, $\{x : f(x) = c\}$ . A contour plot draws several level sets. Introduced in: Lesson 1
Levenberg–Marquardt method: The damped Gauss–Newton step $(J^\top J + \lambda I)h = -J^\top r$ . Small $\lambda$ gives Gauss–Newton, large $\lambda$ a short gradient step. Introduced in: Lesson 10
LICQ: Linear independence constraint qualification: the gradients of the equality constraints and of the active inequality constraints are linearly independent at the point. Introduced in: Lesson 8
line search: A rule that chooses the step length $\alpha$ along a descent direction at each iteration, instead of a fixed step. Introduced in: Lesson 3
linear convergence: The error shrinks by at least a fixed factor $\rho < 1$ per step, $\|e_{k+1}\| \le \rho\|e_k\|$ . On a log scale the error falls along a straight line. Introduced in: Lesson 3
linear scaling rule: When the batch size is multiplied by $c$ , multiply the learning rate by $c$ (Goyal et al. 2017). It holds only below the critical batch size and needs warmup. Introduced in: Lesson 5
local minimum: A point where $f$ is not larger than at any nearby point. It can be higher than the global minimum. Introduced in: Lesson 1
look-ahead point: The extrapolated point $y_k$ where Nesterov's method takes the gradient, one momentum step ahead of $x_k$ . Introduced in: Lesson 4
loss surface: The graph of the loss as a function of the parameters. With two parameters it is a surface over the plane. Introduced in: Lesson 1
majorize–minimize: Minimizing at each step an upper bound of $F$ that touches $F$ at the current point. Then $F$ cannot increase. Introduced in: Lesson 9
minibatch gradient: The average of $B$ example gradients, $\frac1B\sum_{j=1}^B\nabla f_{i_j}(x)$ . It is unbiased, and its variance is $\sigma^2/B$ when the indices are independent. Introduced in: Lesson 5
minimizer: A point $x^\star$ with $f(x^\star) \le f(x)$ for all $x$ (global) or for all $x$ near $x^\star$ (local). The value $f^\star = f(x^\star)$ is the optimal value. Introduced in: Lesson 1
momentum: Adding a fraction $\beta$ of the previous step to the current one. It averages gradients over time, damps the zig-zag and speeds up motion along a valley. Introduced in: Lesson 4
Muon: An optimizer for 2D weight matrices that orthogonalizes the momentum with a few Newton–Schulz steps and uses the result as the update. Other parameters use AdamW. Introduced in: Lesson 12
Nesterov's accelerated gradient method: A momentum method that evaluates the gradient at a look-ahead point $y_k = x_k + \beta_k(x_k - x_{k-1})$ . For every smooth convex function it reaches $f(x_k) - f^\star = O(1/k^2)$ . Introduced in: Lesson 4
Newton's method: The method $x_{k+1} = x_k + \Delta x$ where the Newton step solves $\nabla^2 f(x_k)\,\Delta x = -\nabla f(x_k)$ . It jumps to the minimum of the local quadratic model. Introduced in: Lesson 10
Newton–Schulz iteration: A polynomial iteration $X \leftarrow aX + b(XX^\top)X + c(XX^\top)^2X$ that pushes the singular values of a normalized matrix toward 1 with matrix products only, so it approximates $UV^\top$ . Introduced in: Lesson 12
noise ball: The region around the minimizer where SGD with a fixed step keeps moving. Its size grows with the step size and with the gradient variance $\sigma^2/B$ . Introduced in: Lesson 5
objective (loss): The function $f$ that an optimization method minimizes. In machine learning it is the loss $L(\theta)$ , the average error of the model with parameters $\theta$ . Introduced in: Lesson 1
optimization problem: The task of finding $x$ that makes an objective $f(x)$ as small as possible, written $\min_x f(x)$ . Training a model is an optimization problem over its parameters. Introduced in: Lesson 1
orthogonalized gradient: For a gradient matrix $G = U\Sigma V^\top$ , the matrix $UV^\top$ (the polar factor). It keeps the singular vectors and sets all nonzero singular values to 1. Introduced in: Lesson 12
Polyak–Łojasiewicz (PL) inequality: The inequality $\|\nabla f(x)\|^2 \ge 2\mu\,(f(x) - f^\star)$ . Strong convexity implies it, and it alone gives gradient descent a linear rate, also for some nonconvex functions. Introduced in: Lesson 2
preconditioned sharpness: The largest eigenvalue of the preconditioned Hessian $P^{-1}\nabla^2 L$ , the quantity that limits the stable step of a preconditioned method such as Adam. Introduced in: Lesson 11
probability simplex: The set of vectors with nonnegative entries that sum to 1, $\{x : x \ge 0,\ \mathbf 1^\top x = 1\}$ . Introduced in: Lesson 8
progressive sharpening: The rise of the sharpness during full-batch training while $S < 2/\eta$ . Introduced in: Lesson 11
projected gradient descent: The proximal gradient method with $g$ the indicator of a convex set $C$ : a gradient step, then the projection onto $C$ . Introduced in: Lesson 9
proximal gradient method (ISTA): For $F = f + g$ with $f$ smooth: a gradient step on $f$ , then a prox step on $g$ , $x^+ = \operatorname{prox}_{\eta g}(x - \eta\nabla f(x))$ . With $\eta \le 1/L$ it reaches $F(x_k) - F^\star = O(1/k)$ . Introduced in: Lesson 9
proximal operator: The map $\operatorname{prox}_{\eta g}(v) = \arg\min_x \big(g(x) + \tfrac{1}{2\eta}\|x - v\|^2\big)$ . For an indicator function it is the projection onto the set. Introduced in: Lesson 9
quadratic convergence: Convergence with $\|e_{k+1}\| \le C\|e_k\|^2$ near the solution, so the number of correct digits roughly doubles at every step. Introduced in: Lesson 10
quasi-Newton method: A method that builds an approximation of the Hessian (or of its inverse) from changes of the gradient, without computing second derivatives. Introduced in: Lesson 10
region of confusion: The region where the example gradients point in different directions, so a stochastic step can move away from the minimizer. Far from it, all examples agree. Introduced in: Lesson 5
RMSProp: An adaptive method that divides each coordinate of the gradient by the square root of an EMA of its squared gradients, $\sqrt{v_t} + \varepsilon$ . Introduced in: Lesson 6
saddle point: A stationary point where $f$ goes up in some directions and down in others, for example $(0, 0)$ for $\tfrac12(x^2 - y^2)$ . Introduced in: Lesson 1
saddle-free Newton: A Newton-type method that uses $|H| = Q|\Lambda|Q^\top$ instead of the Hessian, so it moves away from saddle points along negative-curvature directions. Introduced in: Lesson 11
secant condition: The requirement $B_{k+1}s_k = y_k$ (or $H_{k+1}y_k = s_k$ ) with $s_k = x_{k+1} - x_k$ and $y_k = \nabla f(x_{k+1}) - \nabla f(x_k)$ . Introduced in: Lesson 10
Shampoo: A matrix preconditioner with $L = \sum GG^\top$ and $R = \sum G^\top G$ and update $L^{-1/4}GR^{-1/4}$ . With only the current gradient it gives $UV^\top$ . Introduced in: Lesson 12
sharpness: The largest eigenvalue of the Hessian of the loss, $S(\theta) = \lambda_{\max}(\nabla^2 L(\theta))$ . Gradient descent on a quadratic is stable only for $\eta < 2/S$ . Introduced in: Lesson 11
sharpness-aware minimization (SAM): A method that minimizes the worst loss in a ball of radius $\rho$ , $\max_{\|\varepsilon\|_2 \le \rho} L(w + \varepsilon)$ , to find flatter minima. Each step needs two gradients. Introduced in: Lesson 11
sign descent: The update $\theta \leftarrow \theta - \eta\operatorname{sign}(g)$ . Adam's first step, and Adam with $\beta_1 = \beta_2 = 0$ and $\varepsilon = 0$ , is sign descent. Introduced in: Lesson 6
Slater's condition: For a convex problem, there is a strictly feasible point ( $f_i(x) < 0$ for the nonaffine inequalities). It implies strong duality. Introduced in: Lesson 8
SOAP: Adam run in the eigenbasis of Shampoo's matrices $L$ and $R$ ; the eigenbasis is updated only every few steps. Introduced in: Lesson 12
soft thresholding: The proximal operator of $\lambda\|x\|_1$ with step $\eta$ : each entry becomes $\operatorname{sign}(v_i)\max(|v_i| - \eta\lambda, 0)$ . Small entries become exactly zero. Introduced in: Lesson 9
square-root scaling rule: For Adam, when the batch size is multiplied by $c$ , multiply the learning rate by $\sqrt c$ (Malladi et al. 2022). Introduced in: Lesson 7
stationary point: A point with $\nabla f(x) = 0$ (also called a critical point). Minima, maxima and saddle points are stationary points. Introduced in: Lesson 1
steepest descent under a norm: The step that decreases the linear model $\langle g, \Delta\rangle$ most among steps of norm at most 1. Under $\ell_2$ it is $-g/\|g\|$ , under $\ell_\infty$ it is $-\operatorname{sign}(g)$ , under the spectral norm it is $-UV^\top$ . Introduced in: Lesson 12
stochastic gradient descent (SGD): Gradient descent that uses a random, cheap estimate $g_k$ of the gradient (from one example or a minibatch) instead of the full gradient. Introduced in: Lesson 5
strict saddle: A stationary point with $\lambda_{\min}(\nabla^2 f(x)) < 0$ . Gradient descent from a random start almost surely does not converge to a strict saddle. Introduced in: Lesson 11
strictly convex: Convex with strict inequality for $x \ne y$ and $\theta \in (0, 1)$ . It has at most one minimizer, but it need not be strongly convex ( $x^4$ ). Introduced in: Lesson 2
strong duality: The case $d^\star = p^\star$ . It holds for convex problems under Slater's condition, but not for every convex problem. Introduced in: Lesson 8
subgradient: For a convex $g$ , a vector $s$ with $g(y) \ge g(x) + s^\top(y - x)$ for all $y$ . The set of all subgradients at $x$ is the subdifferential $\partial g(x)$ . Introduced in: Lesson 9
sublinear convergence: The error goes to zero more slowly than any fixed factor per step, for example $f(x_k) - f^\star = O(1/k)$ for gradient descent on a convex $L$ -smooth function. Introduced in: Lesson 3
superlinear convergence: Convergence with $\|e_{k+1}\| \le c_k\|e_k\|$ and $c_k \to 0$ . BFGS converges superlinearly near a minimizer. Introduced in: Lesson 10
support vectors: The training points of an SVM with a nonzero dual variable $\alpha_i > 0$ . By complementary slackness they lie on the margin, and only they define $w$ . Introduced in: Lesson 8
total variation: The penalty $\|Dx\|_1$ , with $D$ a difference operator. It favours piecewise-constant signals and images. Introduced in: Lesson 9
transition state: An index-1 saddle point of a potential energy surface (one negative Hessian eigenvalue). It is the highest point of the lowest path between two minima. Introduced in: Lesson 11
trust region: A ball around the current point where the quadratic model is trusted. The step minimizes the model only inside this ball, and the radius changes with the quality of the model. Introduced in: Lesson 10
warm start: Starting a solver from the solution of a nearby problem, for example the previous $\lambda$ on a lasso path. Introduced in: Lesson 9
warmup: A first phase of training where the learning rate grows from almost zero to its peak value, usually linearly over $W$ steps. Introduced in: Lesson 7
warmup–stable–decay (WSD): A schedule with three phases: warmup, a long phase at a constant learning rate, and a short cooldown to a small value. The stable phase can be extended without planning $T$ . Introduced in: Lesson 7
weight decay: Shrinking the weights toward zero at every step, $\theta \leftarrow (1 - \eta\lambda)\theta$ . For plain SGD it is the same as L2 regularization; for Adam it is not. Introduced in: Lesson 6
Wolfe conditions: The Armijo condition plus a curvature condition $\nabla f(x + \alpha p)^\top p \ge c_2\nabla f(x)^\top p$ with $c_1 < c_2 < 1$ . The strong form bounds the absolute value of the new slope. Introduced in: Lesson 3
zig-zag: The path of gradient descent in a narrow valley. The step is limited by the steep direction, so the iterates jump across the valley and move slowly along it. Introduced in: Lesson 1
μ-strongly convex: A function with $f(y) \ge f(x) + \nabla f(x)^\top(y - x) + \tfrac{\mu}{2}\|y - x\|^2$ for some $\mu > 0$ . For a twice differentiable $f$ this means $\nabla^2 f \succeq \mu I$ everywhere. Introduced in: Lesson 2