Start here: notation and Python · 137

Glossary

The technical terms of this track, with short definitions (and the Persian term where available). Each term links to the lesson where it is introduced.

affine function: A function $f(x) = ax + b$ (or $w^\top x + b$ in many inputs). Its graph is a line or a plane; ML calls $w$ the weights and $b$ the bias. Introduced in: Lesson 4
argmax: The input at which a function is largest. $\max_x f(x)$ is the largest value; $\arg\max_x f(x)$ is where it is reached. Introduced in: Lesson 2
average rate of change: The slope $\frac{f(x + h) - f(x)}{h}$ of the secant line through two points of a graph. Introduced in: Lesson 7
axis: One direction of an array. Axis 0 runs down the rows, axis 1 along each row; X.sum(axis=0) adds down the rows and removes axis 0. Introduced in: Lesson 3
base case: The first statement $P(n_0)$ of a proof by induction, checked directly. Introduced in: Lesson 9
boolean mask: An array of True/False values used as an index. x[x > 0] keeps the entries where the condition holds and returns a copy. Introduced in: Lesson 3
broadcasting: NumPy's rule for operations on arrays of different shapes. Shapes are aligned on the right; sizes must be equal or 1, and a size-1 axis is repeated. Introduced in: Lesson 3
Cartesian product: The set $A \times B$ of all ordered pairs $(a, b)$ with $a \in A$ and $b \in B$ . $\R^n$ is the Cartesian product of $n$ copies of $\R$ . Introduced in: Lesson 2
central difference: The estimate $\frac{f(x + h) - f(x - h)}{2h}$ of $f'(x)$ . Its truncation error is of order $h^2$ ; the forward difference has error of order $h$ . Introduced in: Lesson 7
central limit theorem: The average of $n$ independent random variables with the same distribution (mean $\mu$ , standard deviation $\sigma$ ) is close to $\mathcal N(\mu, \sigma^2/n)$ for large $n$ , whatever the shape of the distribution. Introduced in: Lesson 8
chain rule: The rule $(f(g(x)))' = f'(g(x))\,g'(x)$ for the derivative of a composition. Backpropagation applies it once per layer. Introduced in: Lesson 7
codomain: The set $Y$ in $f : X \to Y$ , the type of the outputs. Not every element of $Y$ needs to be an output. Introduced in: Lesson 2
conjecture: A claim that nobody has proved or disproved yet. Introduced in: Lesson 9
contour line: A curve on which a function of two inputs has a constant value (a level set). The gradient is perpendicular to it. Introduced in: Lesson 7
contrapositive: The statement $\neg Q \Rightarrow \neg P$ . It is equivalent to $P \Rightarrow Q$ , so proving it proves the original claim. Introduced in: Lesson 9
converse: The statement $Q \Rightarrow P$ for a claim $P \Rightarrow Q$ . It is a different claim and can be false when the original is true. Introduced in: Lesson 9
corollary: A result that follows quickly from a theorem. Introduced in: Lesson 9
cosine similarity: The number $\cos\theta = \dfrac{x^\top y}{\|x\|\,\|y\|}$ between $-1$ and 1. It measures the angle between two vectors and ignores their lengths. Introduced in: Lesson 6
counterexample: One case where the assumptions of a "for all" claim hold but the conclusion fails. One counterexample is enough to show the claim is false. Introduced in: Lesson 9
critical point: A point $x^*$ with $f'(x^*) = 0$ (or $\nabla f(x^*) = 0$ ). A smooth inner minimum or maximum is one, but a critical point can be neither, as $x^3$ at 0. Introduced in: Lesson 7
cross-entropy loss: The loss $-\log \operatorname{softmax}(z)_y = \operatorname{LSE}(z) - z_y$ for logits $z$ and true class $y$ , the negative log-probability of the true class. Introduced in: Lesson 5
cumulative distribution function: The function $F(x) = P(X \le x)$ . For the standard normal it is written $\Phi(z)$ . Introduced in: Lesson 8
data matrix: The matrix $X \in \R^{m \times n}$ with the $m$ samples as rows and the $n$ features as columns, so $X_{ij} = x^{(i)}_j$ . Introduced in: Lesson 2
derivative: The limit $f'(x) = \lim_{h \to 0} \frac{f(x + h) - f(x)}{h}$ , the slope of the tangent line at $x$ . Introduced in: Lesson 7
desirable difficulty: A difficulty during practice that improves long-term learning, such as recalling instead of re-reading. It helps only if you can still succeed with effort. Introduced in: Lesson 1
direct proof: A proof that starts from the assumptions and reaches the conclusion step by step. Introduced in: Lesson 9
directional derivative: The slope $D_u f = \nabla f^\top u = \|\nabla f\| \cos\theta$ of $f$ in the direction of a unit vector $u$ . Introduced in: Lesson 7
domain: The set $X$ of allowed inputs of a function $f : X \to Y$ . Introduced in: Lesson 2
dot product: The number $x^\top y = \sum_{j=1}^{n} x_j y_j$ for two vectors of the same length. Books also write $x \cdot y$ or $\langle x, y \rangle$ . Introduced in: Lesson 2
dtype: The type of the entries of an array, for example float64 or int64. An integer array stays integer when you store a decimal in it. Introduced in: Lesson 3
dummy variable: An index whose name does not matter. $\sum_{i=1}^{n} x_i$ and $\sum_{j=1}^{n} x_j$ are the same number. Introduced in: Lesson 2
ease: The growth factor of the review interval of a card in SM-2. It starts at 2.5, goes down after Hard or Again (minimum 1.3) and up after Easy. Introduced in: Lesson 1
elementwise product: The product of two arrays of the same shape entry by entry, A * B in NumPy, $(A \odot B)_{ij} = A_{ij} B_{ij}$ . Introduced in: Lesson 3
embedding: A vector that represents an object such as a word, sentence or image, with hundreds or thousands of entries. Similar objects get vectors with a high cosine similarity. Introduced in: Lesson 6
empty sum: A sum with no terms, such as $\sum_{i=1}^{0} x_i$ . Its value is 0 (and the value of an empty product is 1). Introduced in: Lesson 2
Euclidean distance: The distance $\|x - y\|$ between two points $x, y \in \R^n$ , the length of the difference vector. Introduced in: Lesson 6
Euler's number: The constant $e \approx 2.71828$ , the base of the natural exponential $e^x$ and of the natural logarithm. Introduced in: Lesson 4
event: A set of outcomes, a subset of the sample space, for example "the die shows an even number". Introduced in: Lesson 8
expectation: The probability-weighted mean of a random variable, $\mathbb E[X] = \sum_x x\,P(X = x)$ for a discrete $X$ . Introduced in: Lesson 2
exponential function: The function $e^x$ (or $b^x$ with base $b > 0$ ). It is always positive and grows by the same factor for each unit step in $x$ . Introduced in: Lesson 4
feature: One measured property of the samples, for example the area of a house. In a data table it is one column; feature $j$ of sample $i$ is $x^{(i)}_j$ . Introduced in: Lesson 2
forgetting curve: The curve of the chance of recall over time. It drops fast just after learning, then more slowly; each review makes the next drop slower. Introduced in: Lesson 1
function: A rule $f : X \to Y$ that gives each input $x \in X$ exactly one output $f(x) \in Y$ . Introduced in: Lesson 2
gradient: The vector of partial derivatives $\nabla f = (\partial f/\partial x_1, \dots, \partial f/\partial x_n)$ . It points in the direction of steepest ascent. Introduced in: Lesson 7
gradient descent: The method that repeats $x \leftarrow x - \eta \nabla f(x)$ to find a minimum of $f$ . On $f(x) = x^2$ it converges for $0 < \eta < 1$ . Introduced in: Lesson 7
histogram: A bar plot of data grouped into bins. With density heights $\text{count}/(n w)$ for bin width $w$ , the total area of the bars is 1. Introduced in: Lesson 8
illusion of fluency: The feeling that a text is known because it reads easily. Re-reading trains recognition and gives this feeling, but not recall. Introduced in: Lesson 1
inclusion–exclusion: The rule $|A \cup B| = |A| + |B| - |A \cap B|$ for finite sets. The elements of $A \cap B$ are counted twice in $|A| + |B|$ . Introduced in: Lesson 2
independent events: Events $A$ and $B$ with $P(A \text{ and } B) = P(A)\,P(B)$ . Knowing that one happened does not change the chance of the other. Introduced in: Lesson 8
index: The variable that counts the terms of a sum, product or list, as $i$ in $\sum_{i=1}^{n} x_i$ or in $x_i$ . Introduced in: Lesson 2
induction hypothesis: The assumption $P(k)$ used inside the induction step. It is not circular, because the step only proves the implication $P(k) \Rightarrow P(k + 1)$ . Introduced in: Lesson 9
induction step: The part of a proof by induction that shows $P(k) \Rightarrow P(k + 1)$ for every $k \ge n_0$ . Introduced in: Lesson 9
intercept: The value $b$ in $y = ax + b$ , the height of the line at $x = 0$ . Introduced in: Lesson 4
interleaving: Mixing different types of problems in one practice session, instead of doing all problems of one type in a block. Introduced in: Lesson 1
intersection: The set $A \cap B$ of the elements that are in both $A$ and $B$ . Introduced in: Lesson 2
interval (review): The number of days until the next review of a card. On this site, a card rated Good every time comes back after 1, 3, 8, 20, 50 days. Introduced in: Lesson 1
inverse function: The function $f^{-1}$ that undoes $f$ , so $f^{-1}(f(x)) = x$ . It exists when each output comes from exactly one input. Introduced in: Lesson 2
law of large numbers: As the number $n$ of independent samples grows, their average $\bar X$ gets close to the mean $\mu$ . Introduced in: Lesson 8
learning rate: The step size $\eta > 0$ in gradient descent. Too small is slow; too large makes the iterates zig-zag or diverge. Introduced in: Lesson 7
lemma: A smaller result that is proved to help prove a theorem. Introduced in: Lesson 9
linear combination: A sum of scaled vectors, such as $c_1 u + c_2 v$ . Introduced in: Lesson 6
local linear approximation: The estimate $f(x + h) \approx f(x) + f'(x)\,h$ by the tangent line. For smooth $f$ the error shrinks like $h^2$ . Introduced in: Lesson 7
log rules: $\log(ab) = \log a + \log b$ , $\log(a/b) = \log a - \log b$ and $\log(a^k) = k \log a$ , for $a, b > 0$ . Introduced in: Lesson 4
log-likelihood: The log of the likelihood. For independent data points it is $\sum_i \log p_i$ , a sum instead of a product. Training minimizes the negative log-likelihood. Introduced in: Lesson 4
log-log plot: A plot with log scales on both axes. A power law $y = c\,x^k$ is a straight line on it with slope $k$ . Introduced in: Lesson 4
log-softmax: The log of the softmax, $\log \operatorname{softmax}(z)_k = z_k - \operatorname{LSE}(z)$ . Computed this way it is stable for large scores. Introduced in: Lesson 5
log-sum-exp: The function $\operatorname{LSE}(z) = \log \sum_{j=1}^{n} e^{z_j}$ , a smooth maximum with $\max_j z_j \le \operatorname{LSE}(z) \le \max_j z_j + \log n$ . Introduced in: Lesson 5
logarithm: The inverse of the exponential. $\log x$ is the number $y$ with $e^y = x$ , defined for $x > 0$ . In ML $\log$ means the natural log (school books write $\ln$ ). Introduced in: Lesson 4
logit: The log of the odds, $\operatorname{logit}(p) = \log \frac{p}{1 - p}$ . It is the inverse of the sigmoid. Introduced in: Lesson 5
logits: The raw real-valued scores $z_1, \dots, z_n$ that a classifier computes for its classes, before softmax or sigmoid turns them into probabilities. Introduced in: Lesson 5
matrix: A table of numbers $A \in \R^{m \times n}$ with $m$ rows and $n$ columns. $A_{ij}$ is the entry in row $i$ and column $j$ (rows first). Introduced in: Lesson 2
matrix product: The product $AB$ with $(AB)_{ij} = \sum_k A_{ik} B_{kj}$ , A @ B in NumPy. The inner sizes must agree. Introduced in: Lesson 3
max trick: Subtracting $\max_j z_j$ from all scores before exponentiating. By shift invariance it does not change softmax, and it prevents overflow in np.exp. Introduced in: Lesson 5
mean: The average $\bar x = \frac{1}{n} \sum_{i=1}^{n} x_i$ of data. It is the balance point of the data. Introduced in: Lesson 8
mean squared error: The loss $\frac{1}{m} \sum_{i=1}^{m} (\hat y^{(i)} - y^{(i)})^2$ , the average of the squared errors of a model over $m$ samples. Introduced in: Lesson 2
median: The middle value of sorted data (the average of the two middle values if $n$ is even). It is robust to outliers. Introduced in: Lesson 8
multi-label classification: A task in which one input can belong to several classes at once. It uses one sigmoid per class instead of one softmax over all classes. Introduced in: Lesson 5
norm: The Euclidean length $\|x\| = \sqrt{x_1^2 + \dots + x_n^2} = \sqrt{x^\top x}$ of a vector. Introduced in: Lesson 6
normal distribution: The bell-shaped distribution $\mathcal N(\mu, \sigma^2)$ with density $\frac{1}{\sigma\sqrt{2\pi}} e^{-(x - \mu)^2 / (2\sigma^2)}$ . About 68 %, 95 % and 99.7 % of it lie within 1, 2 and 3 standard deviations of $\mu$ . Introduced in: Lesson 8
NumPy array: A grid of numbers of one type (dtype) with a shape, the main data structure of NumPy. Operations on arrays run in fast compiled code. Introduced in: Lesson 3
odds: The ratio $p / (1 - p)$ for a probability $p$ . For example, $p = 0.8$ gives odds 4. Introduced in: Lesson 5
orthogonal: Two vectors $x, y$ are orthogonal (perpendicular) when $x^\top y = 0$ . Then $\|x + y\|^2 = \|x\|^2 + \|y\|^2$ . Introduced in: Lesson 6
partial derivative: The derivative $\partial f / \partial x_j$ of a function of several inputs with respect to $x_j$ , with the other inputs held fixed. Introduced in: Lesson 7
polynomial: A sum of powers with coefficients, $a_0 + a_1 x + \dots + a_d x^d$ . The highest power $d$ with $a_d \ne 0$ is its degree. Introduced in: Lesson 4
power law: A relation $y = c\,x^k$ . Scaling laws of neural networks (loss against model size or data) are approximate power laws. Introduced in: Lesson 4
pretesting: Trying to answer a question before you study the topic. Even a wrong guess helps you remember the right answer later. Introduced in: Lesson 1
probability: A number $P(A)$ between 0 and 1 for each event $A$ , with $P(\Omega) = 1$ and $P(A \text{ or } B) = P(A) + P(B)$ for disjoint events. It is the long-run relative frequency. Introduced in: Lesson 8
probability density: A function $p(x) \ge 0$ with total area 1 under its curve. The probability of an interval is the area under $p$ over that interval. Introduced in: Lesson 8
projection: The "shadow" of $y$ on the line of $x$ , the vector $\dfrac{x^\top y}{\|x\|^2}\,x$ . Its signed length $x^\top y / \|x\|$ is the scalar projection. Introduced in: Lesson 6
proof: A chain of steps from the assumptions to the conclusion, where each step follows from a definition, an earlier result or a rule of logic. Introduced in: Lesson 9
proof by contradiction: A proof that assumes the claim is false and derives something impossible, so the claim must be true. The classic example is the proof that $\sqrt 2$ is irrational. Introduced in: Lesson 9
proof by induction: A proof of $P(n)$ for all $n \ge n_0$ with two parts, the base case $P(n_0)$ and the induction step "if $P(k)$ then $P(k + 1)$ ". Introduced in: Lesson 9
pseudo-random numbers: Numbers made by a deterministic algorithm that look random. The same seed gives the same sequence. Introduced in: Lesson 3
random variable: A number that depends on the outcome of an experiment, such as the face of a die. It is discrete (countable values) or continuous (with a density). Introduced in: Lesson 8
range (image): The set $\{f(x) : x \in X\}$ of the outputs that really occur. It is a subset of the codomain. Introduced in: Lesson 2
recall card: A card with a question on the front and a short answer on the back. You try to recall the answer, then rate how well you did. Introduced in: Lesson 1
reduction: An operation that turns many numbers into fewer, such as sum, mean or max. With axis=k it combines the entries along axis $k$ and removes that axis. Introduced in: Lesson 3
relative frequency: The fraction of trials in which an event happened, for example the number of heads divided by the number of flips. Introduced in: Lesson 8
retention: The chance $R$ that you can recall a fact at a given time. In the simple model of the lesson, $R(t) = e^{-(t - t_0)/S}$ . Introduced in: Lesson 1
retrieval practice: Practice by recall. You answer a question from memory before you see the answer. Introduced in: Lesson 1
robust: A statistic is robust when a few extreme values (outliers) change it little. The median is robust; the mean and the variance are not. Introduced in: Lesson 8
round-off error: The error from storing numbers with finite precision. In a numerical derivative it grows when $h$ is very small, so $h$ must not be tiny. Introduced in: Lesson 7
sample: One example of a data set, for example one house. In a data table it is one row; sample $i$ is written $x^{(i)}$ , which is a label, not a power. Introduced in: Lesson 2
sample space: The set $\Omega$ of all possible outcomes of an experiment, for example $\{1, \dots, 6\}$ for a die. Introduced in: Lesson 8
sample variance: The variance with division by $n - 1$ , $s^2 = \frac{1}{n - 1} \sum_i (x_i - \bar x)^2$ . It is unbiased, $\mathbb E[s^2] = \sigma^2$ , for independent samples. Introduced in: Lesson 8
saturation: The flat parts of the sigmoid for large $|z|$ , where its derivative $\sigma(z)(1 - \sigma(z))$ is close to 0. It is one cause of vanishing gradients. Introduced in: Lesson 5
scalar: A single number, as opposed to a vector. Scalar multiplication $c\,x$ multiplies every entry of $x$ by $c$ . Introduced in: Lesson 6
seed: The starting value of a pseudo-random generator, as in np.random.default_rng(0). Fixing it makes results reproducible. Introduced in: Lesson 3
semilog plot: A plot with a log scale on one axis. Exponential growth $y = c\,b^x$ is a straight line on it when the $y$ -axis is logarithmic. Introduced in: Lesson 4
set: A collection of distinct objects, its elements. Order and repetition do not matter, so $\{1, 2\} = \{2, 1, 1\}$ . Introduced in: Lesson 2
set-builder notation: A way to describe a set by a rule, for example $\{x \in \R : x > 0\}$ , "the set of real $x$ such that $x > 0$ ". Introduced in: Lesson 2
shape: The tuple of the sizes of an array along each axis. A matrix in $\R^{m \times n}$ is an array with shape (m, n). Introduced in: Lesson 3
shift invariance (softmax): The fact $\operatorname{softmax}(z + c\mathbf 1) = \operatorname{softmax}(z)$ for every number $c$ . Only differences of scores matter. Introduced in: Lesson 5
sigmoid: The logistic function $\sigma(z) = \dfrac{1}{1 + e^{-z}}$ . It maps any real number into $(0, 1)$ , with $\sigma(0) = 0.5$ and $\sigma(-z) = 1 - \sigma(z)$ . Introduced in: Lesson 5
slice: An index start:stop:step that picks several entries along an axis. The stop is excluded, and a slice keeps the axis (an integer index removes it). Introduced in: Lesson 3
slope: The change in $y$ per unit change in $x$ along a line, $(y_2 - y_1)/(x_2 - x_1)$ . In $y = ax + b$ it is $a$ . Introduced in: Lesson 4
SM-2: A classic spaced-repetition algorithm. It stores an interval and an ease for each card and multiplies the interval by the ease after each successful review. Introduced in: Lesson 1
softmax: The map $\operatorname{softmax}(z)_k = \dfrac{e^{z_k}}{\sum_j e^{z_j}}$ . It turns a vector of scores into positive probabilities that add up to 1. Introduced in: Lesson 5
softplus: The function $\log(1 + e^x)$ , a smooth version of the ReLU $\max(x, 0)$ . It is the log-sum-exp of $(0, x)$ . Introduced in: Lesson 5
spacing effect: The finding that the same study time spread over several sessions, with gaps between them, gives better long-term memory than one long session. Introduced in: Lesson 1
stability: The number $S$ (in days) in the model $R(t) = e^{-(t - t_0)/S}$ . A larger $S$ means slower forgetting; after $S$ days, $R = e^{-1} \approx 0.37$ . Introduced in: Lesson 1
standard deviation: The square root of the variance, a measure of spread in the unit of the data. np.std divides by $n$ , pandas .std() by $n - 1$ . Introduced in: Lesson 8
strictly increasing: A function with $f(x_1) < f(x_2)$ whenever $x_1 < x_2$ . Applying it does not move the argmax, which is why ML can maximize $\log L$ instead of $L$ . Introduced in: Lesson 4
subset: $A \subseteq B$ means that every element of $A$ is also an element of $B$ . Introduced in: Lesson 2
temperature: A number $T > 0$ in $\operatorname{softmax}(z / T)$ . Small $T$ makes the distribution sharper (towards argmax); large $T$ makes it flatter (towards uniform). Introduced in: Lesson 5
testing effect: The finding that recalling information from memory gives better long-term memory than reading it again. Introduced in: Lesson 1
theorem: An important true claim of the form "if these assumptions hold, then this conclusion holds", together with a proof. Introduced in: Lesson 9
transpose: The operation $x^\top$ that turns a column into a row (and a row into a column). For a matrix, $(A^\top)_{ij} = A_{ji}$ . Introduced in: Lesson 2
underflow: A result too small for the number format, which the computer stores as 0.0. For example, $0.01^{1000}$ is 0.0 in float64. Introduced in: Lesson 4
union: The set $A \cup B$ of the elements that are in $A$ or in $B$ (or in both). Introduced in: Lesson 2
unit vector: A vector with norm 1. For $x \ne 0$ , $x / \|x\|$ is the unit vector in the direction of $x$ (normalizing). Introduced in: Lesson 6
variance: The mean squared deviation from the mean, $\frac{1}{n} \sum_{i=1}^{n} (x_i - \bar x)^2$ . Its unit is the data unit squared. Introduced in: Lesson 8
vector: A list of $n$ numbers $x = (x_1, \dots, x_n)$ , an element of $\R^n$ . In this site's math, vectors are columns. Introduced in: Lesson 2
vectorization: Writing a computation as whole-array operations instead of Python loops. The code is shorter, closer to the math, and much faster. Introduced in: Lesson 3
view: An array that shares memory with another array, as a slice does. Changing a view changes the original array. Introduced in: Lesson 3
z-score: The value $z = (x - \mu)/\sigma$ , the number of standard deviations between $x$ and the mean. For normal $X$ it is standard normal, $\mathcal N(0, 1)$ . Introduced in: Lesson 8
Σ-notation: A short way to write a sum. $\sum_{i=1}^{n} x_i$ means $x_1 + x_2 + \dots + x_n$ ; the index runs from the lower bound to the upper bound. Introduced in: Lesson 2