ML Systems · 164

Glossary

The technical terms of this track, with short definitions (and the Persian term where available). Each term links to the lesson where it is introduced.

1F1B: A pipeline schedule that, after a warm-up, alternates one forward and one backward pass. Same bubble as GPipe, but stage $s$ stores at most $p - s$ micro-batches. Introduced in: Lesson 9
acceptance rate: $\alpha$ : the probability that the target accepts one draft token. The expected tokens per target pass are $(1-\alpha^{\gamma+1})/(1-\alpha)$ . Introduced in: Lesson 12
activation checkpointing: Keeping only some activations (for example the input of each layer) and recomputing the others in the backward pass. Also called gradient checkpointing or rematerialization. Introduced in: Lesson 6
activations: The tensors that the forward pass saves because the backward pass needs them. Their memory grows with batch size and sequence length. Introduced in: Lesson 6
active parameters: The parameters used for one token: the $k$ chosen experts plus the shared parts. The FLOPs per token follow them; the memory follows all parameters. Introduced in: Lesson 10
algorithm bandwidth: algbw = $S/t$ : buffer size divided by the time of the collective, as reported by nccl-tests. Introduced in: Lesson 7
all-gather: A collective in which every rank has one piece and at the end every rank has all pieces. Introduced in: Lesson 7
all-reduce: A collective after which every rank has the sum (or another reduction) of the vectors of all ranks. Introduced in: Lesson 7
all-to-all: A collective in which every rank sends a different piece of its data to every other rank. Used in expert parallelism and Ulysses. Introduced in: Lesson 7
alpha-beta model: The cost model $T = \alpha + n\beta$ for one message of $n$ bytes, with latency $\alpha$ and time per byte $\beta$ (one over the bandwidth). Introduced in: Lesson 7
AOTAutograd: The part of torch.compile that traces forward and backward together ahead of time and builds the backward graph. Introduced in: Lesson 13
arithmetic intensity: The ratio $I$ = FLOPs / bytes moved between HBM and the chip, in FLOP per byte. Introduced in: Lesson 2
autotuning: Choosing kernel parameters such as the tile shape by timing a few candidates and keeping the fastest. Introduced in: Lesson 3
AWQ: Activation-aware weight quantization: scale up the about 1% of input channels of $W$ that see the largest activations before rounding (and the activations down by the same factor), so these channels lose less precision. Introduced in: Lesson 11
bandwidth term: The part of a collective's cost that grows with the bytes sent: $\beta$ times the bytes per rank. Introduced in: Lesson 7
bank conflict: Several threads of a warp access different addresses in the same bank of shared memory. The accesses then run one after the other. Introduced in: Lesson 3
bf16: A 16-bit float with 8 exponent bits and 8 significant bits: the top 16 bits of an fp32 number. Same range as fp32, but low precision. Introduced in: Lesson 5
bisection bandwidth: The bandwidth across a cut that splits the ranks into two halves. It limits all-to-all on a real network. Introduced in: Lesson 7
block table: The map from the logical blocks of a sequence to physical blocks of the KV cache pool. Introduced in: Lesson 12
bus bandwidth: busbw = algbw × $f$ , with $f = 2(p-1)/p$ for all-reduce. For a good algorithm it is about the speed of the slowest link, for any $p$ . Introduced in: Lesson 7
calibration: Choosing quantization ranges from a small set of samples (a few hundred, no labels), for example by min-max, a percentile or the smallest MSE. Introduced in: Lesson 11
capacity factor: The factor CF in the maximum number of tokens per expert, $C = \mathrm{CF} \cdot kN/E$ . Tokens above the capacity are dropped. Introduced in: Lesson 10
chunked prefill: Splitting a long prompt into chunks of a few hundred tokens and running one chunk per iteration next to the decode tokens, so TPOT does not spike. Introduced in: Lesson 12
clipping error: The error of values outside the quantization grid, which move to its end. It has no bound. Introduced in: Lesson 11
coalescing: Neighbouring threads of a warp read neighbouring addresses, so the hardware serves the warp with few memory transactions and every fetched byte is used. Introduced in: Lesson 3
collective: A communication operation that all $p$ ranks call together, such as all-reduce or all-gather. Introduced in: Lesson 7
column split: Splitting $A = [A_1, A_2]$ by columns: GPU $i$ computes $XA_i$ , a block of columns of the output. No communication is needed. Introduced in: Lesson 9
compute-bound: A kernel whose time is set by its arithmetic (FLOPs / $P$ ), because its arithmetic intensity is above the ridge point. Introduced in: Lesson 2
context parallelism (CP): Splitting the whole sequence over GPUs, attention included, for example with ring attention or Ulysses. Introduced in: Lesson 10
continuous batching: Iteration-level scheduling: the scheduler adds and removes requests before every decode step. Introduced in: Lesson 12
copy-on-write: Before a sequence writes into a shared block (reference count above 1), it gets a private copy. Introduced in: Lesson 12
CUDA graphs: Recording a sequence of kernel launches once and replaying it with one launch, so the CPU cost per kernel disappears. Introduced in: Lesson 13
CUDA stream: A queue of GPU work. PyTorch puts kernels into it and returns at once; the GPU runs them in order. Introduced in: Lesson 13
data parallelism: Every GPU holds the whole model and runs a different part of the batch; the gradients are averaged before the update. Introduced in: Lesson 8
DDP: Distributed data parallel: PyTorch's data parallelism, which all-reduces the gradients in buckets during the backward pass. Introduced in: Lesson 8
decode: The phase of LLM inference that generates one token per sequence per step. Each step reads all weights and the KV cache, so it is usually memory-bound. Introduced in: Lesson 2
DeepSpeed-Ulysses: Context parallelism that uses an all-to-all to switch from sequence sharding to head sharding for attention, and back after it. Introduced in: Lesson 10
double binary tree: NCCL's tree algorithm: two binary trees in which no rank is an inner node in both trees; each tree reduces and broadcasts half of the data. Its depth grows like $\lg p$ . Introduced in: Lesson 7
dynamic loss scaling: Loss scaling that changes $S$ during training: skip the step and halve $S$ when a gradient is inf or NaN; double $S$ after a number of good steps. Introduced in: Lesson 5
eager mode: Running each PyTorch operation at once: Python, dispatcher, one kernel. Introduced in: Lesson 13
epilogue: The part of a matmul kernel that applies the bias, the activation or another elementwise operation to each output tile before it is written. Introduced in: Lesson 3
expert parallelism (EP): Putting $E/N_e$ experts on each of $N_e$ GPUs. Tokens go to their experts and back with two all-to-alls. Introduced in: Lesson 10
exposed communication: Communication time that is not hidden behind compute, so the compute stream is empty while NCCL kernels run. Introduced in: Lesson 13
fake quantization: Quantize, then dequantize, in the forward pass; the values keep a float type but lie on the quantization grid. Introduced in: Lesson 11
FlashAttention: An exact attention kernel that computes the output tile by tile with the online softmax. It never writes the $N \times N$ score matrix to HBM, so it moves far fewer bytes and needs $O(N)$ extra memory. Introduced in: Lesson 4
fp16: A 16-bit float with 5 exponent bits and 11 significant bits. More precise than bf16, but its small range needs loss scaling in training. Introduced in: Lesson 5
FP8 (E4M3 and E5M2): Two 8-bit float formats. E4M3 (largest value 448) has more precision and is used for weights and activations; E5M2 (largest value 57,344) has more range and is used for gradients. Introduced in: Lesson 5
FSDP: Fully sharded data parallel: PyTorch's implementation of ZeRO-3. It all-gathers the weights of a unit before its forward and backward, and reduce-scatters the gradients. Introduced in: Lesson 8
FSDP unit: The group of parameters that FSDP gathers and frees together, usually one transformer block. Introduced in: Lesson 8
goodput: The number of requests per second that finish within their latency targets (SLOs). Introduced in: Lesson 12
GPipe: A pipeline schedule that runs all $m$ forward passes, then all $m$ backward passes. Each stage stores the activations of all $m$ micro-batches. Introduced in: Lesson 9
GPTQ: A PTQ method for LLM weights that quantizes one input row at a time and updates the rows not yet quantized to compensate the error, using the Hessian $2X^\top X$ . Introduced in: Lesson 11
gradient accumulation: Running $m$ micro-batches one after the other and adding their gradients before one optimizer step. Introduced in: Lesson 8
gradient bucket: A group of parameters (25 MiB by default in DDP) whose gradients are all-reduced together, as soon as all of them are ready. Introduced in: Lesson 8
graph break: A point where TorchDynamo cannot trace the code (for example .item() or print); the code runs in Python and the graph is split. Introduced in: Lesson 13
grid: All thread blocks of one kernel launch. Introduced in: Lesson 1
group-wise quantization: One scale per group of $g$ consecutive input elements (often 32 to 128) in each column of a weight matrix. Introduced in: Lesson 11
grouped-query attention (GQA): Attention in which $g$ query heads share one key/value head. The KV cache is $g$ times smaller, and the intensity of decode attention grows from about 1 to about $g$ . Introduced in: Lesson 2
guard: A cheap check of what a compiled graph was specialized on (shapes, dtypes, constants). If a guard fails, the function is recompiled. Introduced in: Lesson 13
HBM: High-bandwidth memory: the main memory of the GPU (80 GB on an H100). It is large but far from the SMs, so a load takes hundreds of clock cycles. Introduced in: Lesson 1
HFU: Hardware FLOPs utilization: like MFU, but it counts every FLOP the hardware runs, recomputation included. Introduced in: Lesson 13
hierarchical all-reduce: An all-reduce in three steps: reduce-scatter inside each node over NVLink, all-reduce across nodes, all-gather inside each node. Introduced in: Lesson 7
hierarchical roofline: A roofline with one roof per memory level (HBM, L2, shared memory), each with its own bandwidth. Introduced in: Lesson 2
host-device sync: A point where the CPU waits for the GPU (.item(), .cpu(), print(tensor)), which empties the queue and leaves a gap. Introduced in: Lesson 13
HSDP: Hybrid sharded data parallel: shard inside a group of GPUs (usually one node) and replicate across the groups. Introduced in: Lesson 8
interleaved 1F1B: 1F1B in which each GPU holds $v$ non-adjacent chunks of layers. The bubble shrinks to $\frac{p-1}{vm+p-1}$ , at $v$ times the messages. Introduced in: Lesson 9
Kahan summation: Summation with a second variable that keeps the part lost in each addition and adds it back. The error bound almost does not grow with $n$ . Introduced in: Lesson 5
kernel: A program that runs on the GPU. One kernel launch runs the same code over many threads. Introduced in: Lesson 1
kernel fusion: Joining several operations into one kernel, so the intermediate results stay on the chip and are not written to HBM and read back. Introduced in: Lesson 3
KV cache: The keys and values of all earlier tokens, stored so that decode does not recompute them. Per token it takes $2 \cdot L \cdot n_{kv} \cdot d_h \cdot e$ bytes. Introduced in: Lesson 12
latency hiding: Keeping the GPU busy while some warps wait for memory: the scheduler runs other warps that are ready. Introduced in: Lesson 1
latency term: The part of a collective's cost that grows with the number of messages: $\alpha$ times the number of steps. Introduced in: Lesson 7
launch cost: The fixed time to start a kernel: CPU time to prepare and submit it (a few µs per operation in eager PyTorch) plus a small time on the GPU. Introduced in: Lesson 3
launch-bound: A workload in which the CPU time to launch each kernel is longer than the kernel itself, so the GPU waits between kernels. Introduced in: Lesson 13
Little's law: Data in flight = bandwidth × latency. It says how many bytes must be requested at once to use the full memory bandwidth. Introduced in: Lesson 1
load-balancing loss: An extra loss $\alpha E \sum_i f_i P_i$ that pushes the router to send the same number of tokens to every expert. Introduced in: Lesson 10
logsumexp: $\mathrm{lse}_i = m_i + \log \ell_i = \log \sum_j e^{s_{ij}}$ . FlashAttention stores this one number per row, so the backward pass can recompute the attention probabilities. Introduced in: Lesson 4
loss scaling: Multiplying the loss by a factor $S$ before backward, so that small fp16 gradients do not underflow, and dividing the gradients by $S$ in fp32 before the update. Introduced in: Lesson 5
machine epsilon: $\varepsilon = 2^{1-p}$ : the gap between 1 and the next representable float. Introduced in: Lesson 5
master weights: The fp32 copy of the weights that the optimizer updates. Small updates are not lost to rounding, as they would be in bf16 weights. Introduced in: Lesson 5
memory fragmentation: Free GPU memory that is split into pieces too small to use. Introduced in: Lesson 6
memory-bound: A kernel whose time is set by its memory traffic (bytes / BW), because its arithmetic intensity is below the ridge point. Introduced in: Lesson 2
MFU: Model FLOPs utilization: $\tau F_\text{model} / (n_\text{gpu} P)$ , the fraction of the dense peak that goes into the FLOPs the model needs (no recomputation). Introduced in: Lesson 13
micro-batch: One of the $m$ parts of a batch that go through the pipeline (or through gradient accumulation) one after the other. Introduced in: Lesson 9
mixed-precision training: Training with 16-bit (or 8-bit) matmul inputs, fp32 accumulation, and fp32 master weights and optimizer states. Introduced in: Lesson 5
mixture of experts (MoE): A layer that replaces the MLP by $E$ expert MLPs and a router; each token goes through only $k$ of them. Introduced in: Lesson 10
model states: The memory that belongs to the model: weights, gradients and optimizer states. With mixed-precision Adam it is 16 bytes per parameter. Introduced in: Lesson 6
multi-head latent attention (MLA): DeepSeek's attention that caches one compressed latent vector per token and layer instead of full keys and values per head. Introduced in: Lesson 12
multi-query attention (MQA): Attention with one key/value head shared by all query heads: GQA with one group. Introduced in: Lesson 12
NCCL: The NVIDIA Collective Communications Library. It builds rings and trees over NVLink and the network and picks an algorithm for each call. Introduced in: Lesson 7
NVLink: NVIDIA's fast GPU-to-GPU link inside a node (450 GB/s per direction per GPU on an H100 SXM server, through NVSwitch). Introduced in: Lesson 7
occupancy: The number of resident warps on an SM divided by the maximum number (64 on an H100). Introduced in: Lesson 1
offloading: Moving model states or activations to CPU memory to save GPU memory. It is slow, because the CPU link has low bandwidth. Introduced in: Lesson 6
online softmax: Computing softmax in one pass over blocks: keep a running maximum $m$ , a running sum $\ell$ and a running output, and rescale them when a larger score arrives. The result is exact. Introduced in: Lesson 4
outlier features: A few hidden channels with very large activation values in almost all layers and tokens of large LLMs. They make per-tensor activation quantization fail. Introduced in: Lesson 11
overhead-bound: A workload whose time is set by fixed costs per operation (launch cost, Python), not by FLOPs or bytes. Typical for small tensors. Introduced in: Lesson 3
PagedAttention: vLLM's way to store the KV cache in fixed-size blocks anywhere in a pool, with a block table per sequence, as in virtual memory. Almost no memory is wasted. Introduced in: Lesson 12
pairwise summation: Summing each half recursively and then adding the two results. The error bound grows like $\lceil \log_2 n \rceil\, u$ instead of $n u$ . Introduced in: Lesson 5
per-channel quantization: One scale per output channel (column) of a weight matrix, or one per token for activations. Introduced in: Lesson 11
pipeline bubble: The idle time of the stages at the start and end of a pipeline step: $\frac{p-1}{m+p-1}$ of the step time for GPipe and 1F1B. Introduced in: Lesson 9
pipeline flush: Finishing the backward pass of every micro-batch before the optimizer step, so the update uses exactly the gradient of the whole batch. Introduced in: Lesson 9
pipeline parallelism (PP): Splitting the layers of the model into $p$ consecutive stages on different GPUs; micro-batches flow from stage to stage. Introduced in: Lesson 9
post-training quantization (PTQ): Quantizing a trained model without further training, with scales from the weights and from calibration data. Introduced in: Lesson 11
precision: How fine the grid of representable numbers is. It is set by the number of significant bits $p$ . Introduced in: Lesson 5
preemption: Stopping a running sequence when no KV block is free; its blocks are swapped to CPU memory or recomputed later. Introduced in: Lesson 12
prefetch: Starting the all-gather of the next FSDP unit before the current unit finishes, so communication overlaps with compute. Introduced in: Lesson 8
prefill: The first phase of LLM inference: all tokens of the prompt go through the model together. Its matmuls are usually compute-bound. Introduced in: Lesson 2
prefill/decode disaggregation: Running prefill and decode on separate GPU pools; after the prefill, the KV cache is sent to a decode GPU. Introduced in: Lesson 12
prefix caching: Reusing the cached KV blocks of a common prompt prefix (for example a system prompt) so that its prefill is skipped. Introduced in: Lesson 12
profiler: A tool that records a timeline of the CPU and of each GPU stream, for example Nsight Systems or the PyTorch profiler. Introduced in: Lesson 13
quantization: Storing numbers as small integers $q$ plus a scale: $\hat x = \Delta (q - z)$ . It cuts memory and bytes moved, and can use faster integer or FP8 arithmetic. Introduced in: Lesson 11
quantization-aware training (QAT): Training with fake quantization in the forward pass, so the model learns weights that work well after rounding. Introduced in: Lesson 11
Rabenseifner's algorithm: An all-reduce as a reduce-scatter by recursive halving and an all-gather by recursive doubling. Cost $2\lg p\,\alpha + \frac{2(p-1)}{p} n\beta$ . Introduced in: Lesson 7
range: How large and how small a number a floating-point format can represent. It is set by the exponent bits. Introduced in: Lesson 5
rank: One GPU (or process) that takes part in a collective. The ranks are numbered $0, \dots, p-1$ . Introduced in: Lesson 7
reduce-scatter: A collective after which rank $r$ has the sum, over all ranks, of chunk $r$ only. Introduced in: Lesson 7
rescale factor: The factor $e^{m_\text{old} - m_\text{new}} \le 1$ that the online softmax multiplies the old sum and the old output by when the running maximum grows. Introduced in: Lesson 4
ridge point: The arithmetic intensity $P/\mathrm{BW}$ where the two limits of the roofline meet; about 295 FLOP per byte on an H100 in bf16. Introduced in: Lesson 2
ring all-reduce: An all-reduce as a reduce-scatter then an all-gather around a ring. It costs $2(p-1)\alpha + \frac{2(p-1)}{p} n\beta$ , and its bandwidth term is optimal. Introduced in: Lesson 7
ring attention: Context parallelism in which each GPU keeps its queries and the key/value blocks travel around a ring of GPUs; each GPU merges the partial results with the online softmax. Introduced in: Lesson 10
roofline model: The bound on the speed of a kernel: attainable FLOP/s $= \min(P,\ I \cdot \mathrm{BW})$ , with peak FLOP/s $P$ and memory bandwidth $\mathrm{BW}$ . Introduced in: Lesson 2
round to nearest, ties to even: The default IEEE rounding: round to the nearest float; a value exactly halfway goes to the float whose last bit is 0. Introduced in: Lesson 5
rounding error: The error inside the quantization grid: at most $\Delta/2$ ; about $\Delta^2/12$ in mean square. Introduced in: Lesson 11
router: The small layer of an MoE that computes $p = \mathrm{softmax}(xW_r)$ and chooses the experts for each token. Introduced in: Lesson 10
row split: Splitting $A$ by rows and $X$ by columns: GPU $i$ computes the partial sum $X_iA_i$ , and an all-reduce gives $Y = \sum_i X_iA_i$ . Introduced in: Lesson 9
scale (step size): The real number $\Delta$ that gives the size of one integer step in quantization. Introduced in: Lesson 11
scale factor (FP8): A per-tensor (or per-block) number $\sigma$ that maps a tensor into the range of an FP8 format, for example $\sigma = 448/\mathrm{amax}(x)$ for E4M3. Introduced in: Lesson 5
selective activation recomputation: Keeping all activations except those of the attention core, and recomputing the core from $Q$ , $K$ and $V$ in the backward pass. Introduced in: Lesson 6
sequence parallelism (SP): Megatron's addition to tensor parallelism: the token-wise parts (LayerNorm, dropout) are split along the sequence. It saves activation memory and adds no communication volume. Introduced in: Lesson 10
shared memory: Fast on-chip memory inside each SM. The threads of one block use it to share data and to reuse tiles loaded from HBM. Introduced in: Lesson 1
significant bits: The $p$ bits of the significand of a float, including the leading 1 that is not stored. bf16 has $p = 8$ , fp16 has $p = 11$ , fp32 has $p = 24$ . Introduced in: Lesson 5
SIMT: Single instruction, multiple threads: NVIDIA's execution model, in which the 32 threads of a warp run the same instruction on different data. Introduced in: Lesson 1
SmoothQuant: Moving the difficulty of activation outliers into the weights with an exact rescaling $XW = (X\,\mathrm{diag}(s)^{-1})(\mathrm{diag}(s)\,W)$ . Introduced in: Lesson 11
speculative decoding: A small draft model proposes $\gamma$ tokens and the target model checks all of them in one forward pass. With the rejection rule, the output has exactly the target distribution. Introduced in: Lesson 12
split-KV: A decode attention method that cuts the KV cache of each sequence into $k$ chunks, one thread block per chunk, and merges the partial results with the logsumexp. It keeps all SMs busy at small batch. Introduced in: Lesson 4
static batching: Starting a batch of requests together and running it until its longest request ends. Introduced in: Lesson 12
straggler: One rank that is slower than the others; all others wait for it inside the collectives. Introduced in: Lesson 13
straight-through estimator (STE): The gradient rule of QAT: use $\partial \hat x/\partial x = 1$ inside the clipping range and 0 outside, because rounding has zero derivative almost everywhere. Introduced in: Lesson 11
streaming multiprocessor (SM): One of the many processors of a GPU (132 on an H100). It runs thread blocks and has its own registers, shared memory and tensor cores. Introduced in: Lesson 1
subnormal number: A number below the smallest normal number of a format. It has fewer significant bits, so its relative error is larger. Introduced in: Lesson 5
tail effect: The loss when the last wave of a launch has only a few blocks, so most SMs are idle. Also called wave quantization. Introduced in: Lesson 1
tensor core: A unit inside an SM that computes small matrix products very fast, about 15 times faster than the ordinary cores. Introduced in: Lesson 1
Tensor Memory Accelerator (TMA): A unit of Hopper GPUs that copies whole tiles between global and shared memory, started by one thread. Introduced in: Lesson 3
tensor parallelism (TP): Splitting the weight matrices of each layer over $t$ GPUs, so every GPU computes part of every layer. Introduced in: Lesson 9
TF32: A tensor-core format with the range of fp32 (8 exponent bits) and 10 mantissa bits. PyTorch uses it for fp32 matmuls when allowed. Introduced in: Lesson 5
thread: The smallest unit of work in a kernel. Each thread runs the kernel code on its own part of the data. Introduced in: Lesson 1
thread block: A group of up to 1,024 threads that runs on one SM. Threads of one block can share data in shared memory and wait for each other; threads of different blocks cannot. Introduced in: Lesson 1
throughput: The total work done per second. A GPU is built for throughput, not for the low latency of one piece of work. Introduced in: Lesson 1
tiling: Loading a block (tile) of data into fast on-chip memory once and using it many times. This is how matrix multiplies reach the peak FLOP/s. Introduced in: Lesson 3
top-k routing: Sending each token to the $k$ experts with the largest router probabilities. Introduced in: Lesson 10
TorchDynamo: The part of torch.compile that captures PyTorch operations from Python bytecode into FX graphs, with guards. Introduced in: Lesson 13
TorchInductor: The default backend of torch.compile: it fuses operations and writes Triton kernels on GPUs. Introduced in: Lesson 13
TPOT: Time per output token (inter-token latency): the time between two later tokens of one sequence, about one decode step. Introduced in: Lesson 12
Triton: A Python language for GPU kernels in which one program instance computes one tile. torch.compile generates Triton code. Introduced in: Lesson 3
TTFT: Time to first token: the time in the queue plus the prefill. Introduced in: Lesson 12
unit roundoff: $u = \varepsilon/2 = 2^{-p}$ : the bound on the relative error of one rounding, $\mathrm{fl}(x) = x(1+\delta)$ with $|\delta| \le u$ . Introduced in: Lesson 5
W4A16, W8A8: Names for quantization schemes: W4A16 = 4-bit weights and 16-bit activations; W8A8 = 8-bit weights and 8-bit activations. Introduced in: Lesson 11
warp: A group of 32 threads that the hardware runs together, one instruction at a time for all 32. Introduced in: Lesson 1
warp divergence: When threads of one warp take different branches. The warp runs both branches one after the other, with part of its threads masked off. Introduced in: Lesson 1
wave: The set of thread blocks that run on the GPU at the same time. A launch with more blocks runs in several waves. Introduced in: Lesson 1
ZeRO: Zero Redundancy Optimizer: each data-parallel GPU keeps only its slice of the optimizer states (stage 1), also of the gradients (stage 2), and also of the weights (stage 3). Introduced in: Lesson 8
zero-bubble schedule: A pipeline schedule that splits the backward pass into the input-gradient part and the weight-gradient part, and uses the weight-gradient work to fill the bubble. Introduced in: Lesson 9
zero-point: The integer $z$ that represents the real value 0 in asymmetric quantization. Introduced in: Lesson 11
zig-zag layout: Giving each rank one early and one late chunk of the sequence, so causal attention has the same work on every rank. Introduced in: Lesson 10