learn.mohebi.ai

    Type at least 2 letters. Enter opens the first result, Esc closes.
    197 lessons · 15 tracks

    Concept map

    Every lesson of every track, and what it builds on. Point at a lesson (or tap it) to see what it needs and what comes after it; click it to open it.

    Done Cards due Needed first Builds on it Drag to move. Pinch, or Ctrl + scroll, to zoom.
    Show the map as a list

    Start here: notation and Python

    1. How to learn here: active learning and spaced repetition
    2. Reading math notation: sets, functions, indices, sums and products · Needs: How to learn here
    3. Python and NumPy for math: arrays, broadcasting, plots and random numbers · Needs: Reading math notation
    4. Functions you meet in ML: lines, powers, exp and log · Needs: Python and NumPy for math, Reading math notation
    5. From scores to probabilities: sigmoid, softmax and log-sum-exp · Needs: Lines, powers, exp and log, Reading math notation, Python and NumPy for math
    6. A gentle tour of vectors: arrows, lists, length and the dot product · Needs: Reading math notation, Python and NumPy for math
    7. Rates of change: slope, derivative and the gradient · Needs: A gentle tour of vectors, Lines, powers, exp and log, Python and NumPy for math
    8. Chance and data: probability, mean, variance and the normal curve · Needs: Reading math notation, Python and NumPy for math, Lines, powers, exp and log
    9. How to read and write a proof · Needs: Chance and data, Reading math notation, Sigmoid, softmax, log-sum-exp

    Linear Algebra

    1. Linear algebra in plain words: a tour of the core objects
    2. Matrices as maps, span, basis and the views of matrix multiplication
    3. Rank, rank–nullity and the four fundamental subspaces · Needs: Matrices as maps, span and basis
    4. Inverse, determinant, dot product and orthogonal projection · Needs: Rank and the four subspaces, Matrices as maps, span and basis
    5. Orthogonal and unitary matrices, change of basis, similarity · Needs: Inverse, determinant and projection, Matrices as maps, span and basis
    6. Eigenvalues, eigenvectors, power iteration, diagonalization and Jordan form · Needs: Orthogonal matrices and change of basis, Inverse, determinant and projection
    7. Spectral theorem, Rayleigh quotient, Courant–Fischer and Weyl · Needs: Eigenvalues and Jordan form, Orthogonal matrices and change of basis, Inverse, determinant and projection
    8. Positive (semi-)definite matrices, the five tests, Cholesky, Schur complements and the Loewner order · Needs: Spectral theorem and Rayleigh quotient, Inverse, determinant and projection
    9. The SVD, Eckart–Young–Mirsky, polar decomposition and the pseudoinverse · Needs: Positive definite matrices and Cholesky, Spectral theorem and Rayleigh quotient, Rank and the four subspaces
    10. PCA as SVD of centered data, and randomized SVD · Needs: The SVD, Spectral theorem and Rayleigh quotient
    11. The factorization big picture: LU, QR, QΛQ⊤, XΛX−1, UΣV⊤ · Needs: Eigenvalues and Jordan form, Positive definite matrices and Cholesky, The SVD
    12. The norm zoo: ℓp, spectral, Frobenius, nuclear and Schatten norms · Needs: The SVD, Inverse, determinant and projection
    13. QR (Gram–Schmidt, Householder, Givens) and least squares done right · Needs: Inverse, determinant and projection, Orthogonal matrices and change of basis, The SVD
    14. Floating point, cancellation, mixed precision, conditioning and backward error · Needs: QR and least squares, The SVD, Norms of vectors and matrices
    15. Eigenvalue and SVD algorithms (QR algorithm, Golub–Kahan) and Krylov methods · Needs: Floating point and conditioning, Eigenvalues and Jordan form, Spectral theorem and Rayleigh quotient, Positive definite matrices and Cholesky, QR and least squares
    16. Linear algebra of modern ML: autodiff, LoRA/GaLore, tensor decompositions, attention · Needs: Matrices as maps, span and basis, Rank and the four subspaces, The SVD
    17. Matrix differentials, matrix functions, Kronecker product and vec · Needs: Autodiff, LoRA, tensors, attention, Inverse, determinant and projection, Eigenvalues and Jordan form, Spectral theorem and Rayleigh quotient
    18. Perturbation theory: Weyl and Davis–Kahan · Needs: Spectral theorem and Rayleigh quotient, The SVD, PCA and randomized SVD, Floating point and conditioning
    19. Sketching, Johnson–Lindenstrauss and random matrix theory · Needs: Perturbation theory, PCA and randomized SVD, QR and least squares
    20. Structured matrices in ML for science: circulants and FNO, spectral graphs and GNNs, equivariance, matrix manifolds · Needs: Rank and the four subspaces, Orthogonal matrices and change of basis, Spectral theorem and Rayleigh quotient, The SVD

    Calculus for ML

    1. Derivatives in one variable: slope, linear approximation and the rules
    2. Partial derivatives, the gradient and directional derivatives · Needs: Derivatives in one variable, Matrices as maps, span and basis
    3. The derivative as a linear map: Jacobians, Hessians and Taylor's theorem · Needs: Partial derivatives and the gradient, Matrices as maps, span and basis, Spectral theorem and Rayleigh quotient, Positive definite matrices and Cholesky
    4. The chain rule: products of Jacobians and computational graphs · Needs: Jacobians, Hessians and Taylor, Derivatives in one variable, Partial derivatives and the gradient, Matrices as maps, span and basis
    5. Automatic differentiation: forward mode and reverse mode · Needs: Chain rule and computational graphs, Autodiff, LoRA, tensors, attention
    6. Backpropagation by hand: MLPs, losses and layer normalization · Needs: Forward and reverse mode autodiff, Jacobians, Hessians and Taylor, Chain rule and computational graphs, Matrices as maps, span and basis, Matrix calculus and matrix functions
    7. Numerical differentiation and gradient checking · Needs: Backprop: MLPs, losses, layers, Jacobians, Hessians and Taylor, Floating point and conditioning
    8. Hessian-vector products, Gauss-Newton and trace estimation · Needs: Finite differences and gradient checks, Jacobians, Hessians and Taylor, Forward and reverse mode autodiff, Backprop: MLPs, losses, layers, Spectral theorem and Rayleigh quotient, Positive definite matrices and Cholesky, Eigenvalue algorithms and Krylov methods
    9. The implicit function theorem and gradients through solvers · Needs: Hessian-vector products, Chain rule and computational graphs, Forward and reverse mode autodiff, Inverse, determinant and projection, Floating point and conditioning, Eigenvalue algorithms and Krylov methods
    10. Integrals, expectations and the change of variables · Needs: Derivatives in one variable, Jacobians, Hessians and Taylor, Inverse, determinant and projection
    11. Gaussian integrals and Laplace's method · Needs: Integrals and change of variables, Jacobians, Hessians and Taylor, Spectral theorem and Rayleigh quotient, Positive definite matrices and Cholesky
    12. Differentiating expectations: score-function and pathwise gradients · Needs: Chain rule and computational graphs, Forward and reverse mode autodiff, Integrals and change of variables
    13. Calculus of variations, Euler–Lagrange and physics-informed losses · Needs: Partial derivatives and the gradient, Jacobians, Hessians and Taylor, Integrals and change of variables, Positive definite matrices and Cholesky, Floating point and conditioning

    Probability and Statistics

    1. Probability, conditional probability and Bayes' rule · Needs: Reading math notation, Chance and data
    2. Random variables: PMF, PDF, CDF, expectation and variance · Needs: Probability and Bayes' rule, Integrals and change of variables
    3. Common distributions, the Poisson process and moment generating functions · Needs: Random variables and expectation, Probability and Bayes' rule, Integrals and change of variables
    4. Joint and conditional distributions, conditional expectation and covariance · Needs: Common distributions, Random variables and expectation, Spectral theorem and Rayleigh quotient, Positive definite matrices and Cholesky
    5. Change of variables and the multivariate Gaussian · Needs: Joint distributions and covariance, Common distributions, Integrals and change of variables, Inverse, determinant and projection, Positive definite matrices and Cholesky
    6. The law of large numbers, the central limit theorem and confidence intervals · Needs: Random variables and expectation, Common distributions, Joint distributions and covariance
    7. Concentration inequalities: Chernoff, Hoeffding and sample sizes · Needs: LLN, CLT and confidence intervals, Common distributions, Jacobians, Hessians and Taylor
    8. Maximum likelihood, the exponential family and properties of estimators · Needs: Common distributions, Joint distributions and covariance, LLN, CLT and confidence intervals, Partial derivatives and the gradient, Jacobians, Hessians and Taylor, Jensen and KL divergence
    9. Fisher information, the Cramér–Rao bound and asymptotic normality of the MLE · Needs: Maximum likelihood, LLN, CLT and confidence intervals, Joint distributions and covariance, Jacobians, Hessians and Taylor
    10. Bayesian inference: priors, posteriors, MAP and prediction · Needs: Fisher information, Probability and Bayes' rule, Common distributions, The multivariate Gaussian, Maximum likelihood
    11. Hypothesis testing: p-values, power, t-tests and multiple testing · Needs: Bayesian inference, Probability and Bayes' rule, Random variables and expectation, LLN, CLT and confidence intervals
    12. The bootstrap, permutation tests and A/B testing · Needs: Hypothesis testing, Joint distributions and covariance, LLN, CLT and confidence intervals
    13. Linear regression from the statistics view · Needs: Joint distributions and covariance, The multivariate Gaussian, Maximum likelihood, Bayesian inference, Hypothesis testing, QR and least squares, Inverse, determinant and projection
    14. Monte Carlo: rejection sampling, importance sampling and variance reduction · Needs: Random variables and expectation, Common distributions, LLN, CLT and confidence intervals, Bayesian inference, Sigmoid, softmax, log-sum-exp
    15. Markov chains and MCMC: Metropolis–Hastings, Gibbs and Hamiltonian Monte Carlo · Needs: Monte Carlo and importance sampling, Joint distributions and covariance, The multivariate Gaussian, Bayesian inference, Eigenvalues and Jordan form

    Optimization

    1. Loss surfaces and gradient descent · Needs: Partial derivatives and the gradient, Matrices as maps, span and basis
    2. Convexity, smoothness and the condition number · Needs: Loss surfaces and gradient descent, Jacobians, Hessians and Taylor, Spectral theorem and Rayleigh quotient, Positive definite matrices and Cholesky
    3. Convergence rates of gradient descent and line search · Needs: Convexity, smoothness, conditioning, Loss surfaces and gradient descent, Spectral theorem and Rayleigh quotient
    4. Momentum and Nesterov acceleration · Needs: GD rates and line search, Eigenvalues and Jordan form, Eigenvalue algorithms and Krylov methods
    5. Stochastic gradient descent, the noise ball and batch size · Needs: GD rates and line search, Convexity, smoothness, conditioning
    6. Adaptive methods: AdaGrad, RMSProp, Adam and AdamW · Needs: SGD and batch size, Momentum and Nesterov, Convexity, smoothness, conditioning
    7. Learning-rate schedules, warmup and gradient clipping in practice · Needs: AdaGrad, RMSProp, Adam, AdamW, SGD and batch size, GD rates and line search, Convexity, smoothness, conditioning
    8. Lagrange multipliers, KKT conditions and duality · Needs: Convexity, smoothness, conditioning, Loss surfaces and gradient descent, Partial derivatives and the gradient, Inverse, determinant and projection, Norms of vectors and matrices
    9. Proximal methods (ISTA, FISTA) and ADMM · Needs: Lagrange multipliers, KKT, duality, GD rates and line search, Momentum and Nesterov, Norms of vectors and matrices
    10. Newton, quasi-Newton and Gauss–Newton methods · Needs: Convexity, smoothness, conditioning, GD rates and line search, Positive definite matrices and Cholesky, QR and least squares, Jacobians, Hessians and Taylor
    11. Nonconvex landscapes: saddle points, sharpness and the edge of stability · Needs: Newton, BFGS and Gauss–Newton, Convexity, smoothness, conditioning, GD rates and line search, SGD and batch size, Schedules, warmup and clipping, Eigenvalues and Jordan form, Eigenvalue algorithms and Krylov methods, Hessian-vector products
    12. Matrix optimizers: K-FAC, Shampoo and Muon · Needs: AdaGrad, RMSProp, Adam, AdamW, Newton, BFGS and Gauss–Newton, The SVD, Norms of vectors and matrices, Matrix calculus and matrix functions

    Information Theory

    1. Surprise, bits and entropy · Needs: Lines, powers, exp and log
    2. Joint and conditional entropy, the chain rule · Needs: Surprise, bits and entropy, Joint distributions and covariance
    3. Jensen's inequality and the KL divergence · Needs: Joint and conditional entropy, Surprise, bits and entropy, Convexity, smoothness, conditioning
    4. Cross-entropy: the loss of classifiers and language models · Needs: Jensen and KL divergence, Surprise, bits and entropy, Joint and conditional entropy, Backprop: MLPs, losses, layers
    5. Source coding: compression is prediction · Needs: Cross-entropy as a loss, Surprise, bits and entropy, Joint and conditional entropy, Jensen and KL divergence
    6. Continuous variables: differential entropy, Gaussians and forward vs reverse KL · Needs: Surprise, bits and entropy, Jensen and KL divergence, Cross-entropy as a loss, The multivariate Gaussian, Integrals and change of variables, Inverse, determinant and projection, Positive definite matrices and Cholesky
    7. Mutual information and the data processing inequality · Needs: Differential entropy and KL fits, Joint and conditional entropy, Jensen and KL divergence, Joint distributions and covariance, The multivariate Gaussian
    8. Critics and variational bounds: f-divergences, InfoNCE and contrastive learning · Needs: Mutual information, Jensen and KL divergence, Cross-entropy as a loss, Differential entropy and KL fits, Convexity, smoothness, conditioning
    9. The ELBO: variational inference, VAEs and importance-weighted bounds · Needs: Jensen and KL divergence, Cross-entropy as a loss, Differential entropy and KL fits, Probability and Bayes' rule, Gradients of expectations
    10. Lossy compression: rate–distortion, the information bottleneck and the β-VAE · Needs: The ELBO and VAEs, Compression is prediction, Mutual information, Joint and conditional entropy, Differential entropy and KL fits, PCA and randomized SVD

    Deep Learning

    1. MLPs and universal approximation · Needs: Matrices as maps, span and basis, Rank and the four subspaces
    2. Training a network: losses, mini-batches and the training loop · Needs: MLPs and universal approximation, Eigenvalues and Jordan form
    3. Backpropagation on computational graphs · Needs: Training a network, MLPs and universal approximation, Autodiff, LoRA, tensors, attention, Floating point and conditioning
    4. Activations and initialization from variance propagation · Needs: Backpropagation, MLPs and universal approximation, The SVD
    5. Normalization layers and residual connections · Needs: Activations and initialization, Backpropagation, Inverse, determinant and projection
    6. Regularization: dropout, weight decay, label smoothing, augmentation, early stopping · Needs: Normalization and residuals, Training a network, Backpropagation, The SVD
    7. Convolutional networks: filters, arithmetic, im2col and ResNet · Needs: Backpropagation, Normalization and residuals, Structured matrices in ML
    8. Recurrent networks, LSTMs and vanishing gradients · Needs: Backpropagation, Activations and initialization, Eigenvalues and Jordan form, Norms of vectors and matrices
    9. Attention: queries, keys, values and masks · Needs: RNNs and LSTMs, Backpropagation, Normalization and residuals, Matrix calculus and matrix functions
    10. Positional encodings: sinusoidal, learned, RoPE and ALiBi · Needs: Attention, Orthogonal matrices and change of basis
    11. The transformer: the block, model families and vision transformers · Needs: Positional encodings, Normalization and residuals, Convolutional networks, RNNs and LSTMs, Attention
    12. Training dynamics: warmup, clipping, loss spikes, large batches, mixed precision · Needs: Transformers and ViT, Training a network, Normalization and residuals, RNNs and LSTMs, Attention, Floating point and conditioning
    13. Graph neural networks: message passing, GCN and expressivity · Needs: MLPs and universal approximation, Attention, Eigenvalues and Jordan form, Structured matrices in ML
    14. Self-supervised learning: contrastive, non-contrastive and masked modelling · Needs: Attention, Convolutional networks
    15. Mixture of experts: routing, load balancing and capacity · Needs: Activations and initialization, Normalization and residuals, Attention, Transformers and ViT, Training dynamics
    16. State-space models: from linear recurrences to S4 and Mamba · Needs: Convolutional networks, RNNs and LSTMs, Attention, Matrix calculus and matrix functions, Structured matrices in ML, Eigenvalues and Jordan form
    17. Generalization: double descent and implicit regularization · Needs: Training a network, Regularization, Rank and the four subspaces, The SVD, Sketching, JL and random matrices

    Generative Models

    1. Generative models, latent variables and the ELBO · Needs: Jensen and KL divergence, Differential entropy and KL fits, The ELBO and VAEs
    2. Variational autoencoders: reparameterization, posterior collapse, β-VAE and VQ-VAE · Needs: Latent variables and the ELBO, Gradients of expectations, Rate–distortion and β-VAE, Backpropagation
    3. Autoregressive models: the chain rule, MADE, PixelCNN and transformer language models · Needs: Latent variables and the ELBO, Cross-entropy as a loss, Convolutional networks, Attention
    4. Normalizing flows: change of variables, coupling, autoregressive and continuous flows · Needs: Autoregressive models, Latent variables and the ELBO, Integrals and change of variables, Jacobians, Hessians and Taylor, Hessian-vector products, Inverse, determinant and projection
    5. GANs and Wasserstein GANs: the optimal discriminator, training dynamics and W1 · Needs: Normalizing flows, Latent variables and the ELBO, Jensen and KL divergence, InfoNCE and contrastive learning, Lagrange multipliers, KKT, duality, SGD and batch size, Norms of vectors and matrices
    6. Energy-based models, score matching and Langevin dynamics · Needs: GANs and Wasserstein GANs, Latent variables and the ELBO, Loss surfaces and gradient descent
    7. Denoising diffusion (DDPM): forward process, forward posterior, ELBO and the noise-prediction loss · Needs: EBMs, score matching, Langevin, Latent variables and the ELBO, Variational autoencoders, The multivariate Gaussian
    8. Diffusion parameterizations: ε, x0, v and the score; SNR weighting; EDM preconditioning · Needs: Denoising diffusion (DDPM), EBMs, score matching, Langevin, Integrals and change of variables
    9. Reverse SDEs, the probability-flow ODE and samplers · Needs: Diffusion parameterizations, EBMs, score matching, Langevin, Denoising diffusion (DDPM), Time stepping and stiffness
    10. Flow matching and rectified flow · Needs: Reverse SDEs, PF-ODE and samplers, Normalizing flows, Diffusion parameterizations
    11. Guidance: classifier guidance and classifier-free guidance · Needs: Diffusion parameterizations, Reverse SDEs, PF-ODE and samplers, Bayesian inference
    12. Few-step generation: progressive distillation, consistency models and adversarial distillation · Needs: Guidance, Diffusion parameterizations, Reverse SDEs, PF-ODE and samplers, Flow matching and rectified flow
    13. Latent diffusion and diffusion transformers: autoencoders, the scale factor, cross-attention and DiT · Needs: Variational autoencoders, Diffusion parameterizations, Guidance, Attention, Transformers and ViT, Normalization and residuals
    14. Evaluating generative models: bits per dimension, FID, KID, precision and recall, CRPS and calibration · Needs: Latent variables and the ELBO, Normalizing flows, The multivariate Gaussian, Matrix calculus and matrix functions, Jensen and KL divergence, Probabilistic forecasting, Gaussian processes and UQ
    15. Diffusion models for inverse problems and data assimilation · Needs: Evaluating generative models, Guidance, Diffusion parameterizations, Reverse SDEs, PF-ODE and samplers, EBMs, score matching, Langevin, The multivariate Gaussian, Data assimilation

    Reinforcement Learning

    1. Markov decision processes, returns and the Bellman equations · Needs: Reading math notation, Chance and data, Random variables and expectation, Joint distributions and covariance, Inverse, determinant and projection, Norms of vectors and matrices
    2. Dynamic programming: policy evaluation, policy iteration and value iteration · Needs: MDPs and Bellman equations, Norms of vectors and matrices, Reading and writing proofs
    3. Exploration and multi-armed bandits: ε-greedy, UCB and Thompson sampling · Needs: MDPs and Bellman equations, Chance and data, Concentration inequalities, Bayesian inference
    4. Learning values from experience: Monte Carlo and temporal-difference prediction · Needs: Bandits and exploration, MDPs and Bellman equations, Policy and value iteration, Monte Carlo and importance sampling
    5. TD control: SARSA, Q-learning and the maximization bias · Needs: Monte Carlo and TD prediction, Policy and value iteration, Bandits and exploration, Jensen and KL divergence
    6. Function approximation, DQN and learning from fixed data · Needs: SARSA and Q-learning, Monte Carlo and TD prediction, MLPs and universal approximation, Training a network, SGD and batch size, Partial derivatives and the gradient
    7. Policy gradients: the policy gradient theorem, REINFORCE and baselines · Needs: MDPs and Bellman equations, Monte Carlo and TD prediction, Gradients of expectations, Chain rule and computational graphs, Sigmoid, softmax, log-sum-exp, SGD and batch size
    8. Actor-critic methods: advantages, GAE, A2C and soft actor-critic · Needs: Policy gradients and REINFORCE, Monte Carlo and TD prediction, Function approximation and DQN, SARSA and Q-learning, Surprise, bits and entropy, Gradients of expectations
    9. Trust regions, PPO and RL for language models · Needs: Actor-critic, GAE and SAC, Policy gradients and REINFORCE, Monte Carlo and TD prediction, Jensen and KL divergence, Differential entropy and KL fits, Newton, BFGS and Gauss–Newton
    10. Models, planning and imitation: Dyna, MCTS and AlphaZero · Needs: Policy and value iteration, Bandits and exploration, SARSA and Q-learning, Policy gradients and REINFORCE, Cross-entropy as a loss

    ML for Science

    1. Matrix calculus and reverse-mode automatic differentiation · Needs: Autodiff, LoRA, tensors, attention, Matrix calculus and matrix functions, Chain rule and computational graphs, Forward and reverse mode autodiff, Finite differences and gradient checks
    2. Time stepping: accuracy, stability and stiffness · Needs: Eigenvalues and Jordan form, Matrix calculus and matrix functions, Jacobians, Hessians and Taylor, Finite differences and gradient checks
    3. Adjoint methods for ODEs and neural ODEs · Needs: Time stepping and stiffness, Reverse-mode autodiff, Implicit differentiation, Normalization and residuals
    4. Inverse problems, regularization and seismic imaging · Needs: Adjoints and neural ODEs, The SVD, QR and least squares, Floating point and conditioning, Newton, BFGS and Gauss–Newton
    5. Chaos, predictability, Koopman, DMD and SINDy · Needs: Eigenvalues and Jordan form, The SVD, QR and least squares, Time stepping and stiffness, Adjoints and neural ODEs, Proximal methods and ADMM
    6. Data assimilation: Kalman filter, ensemble Kalman filter and 4D-Var · Needs: Chaos, DMD and SINDy, Adjoints and neural ODEs, Positive definite matrices and Cholesky, State space and Kalman filter
    7. Gaussian processes, scoring rules and calibration · Needs: Data assimilation, Positive definite matrices and Cholesky, Reverse-mode autodiff, Probabilistic forecasting
    8. Fourier spectral methods and Fourier neural operators · Needs: Structured matrices in ML, Time stepping and stiffness
    9. Spherical harmonics and spherical Fourier neural operators · Needs: Spectral methods and FNO, Structured matrices in ML, Gaussian integrals and Laplace
    10. Symmetry, group representations and E(3)-equivariant networks · Needs: Spherical harmonics and SFNO, Orthogonal matrices and change of basis, Structured matrices in ML, Graph neural networks
    11. Physics-informed neural networks and their failure modes · Needs: Reverse-mode autodiff, Time stepping and stiffness, Spectral methods and FNO, MLPs and universal approximation, Training a network, Convexity, smoothness, conditioning
    12. Stochastic differential equations, Itô calculus and the Fokker–Planck equation · Needs: Time stepping and stiffness, Jacobians, Hessians and Taylor, The multivariate Gaussian, Monte Carlo and importance sampling, Markov chains and MCMC
    13. Score matching, the reverse-time SDE and the probability-flow ODE · Needs: SDEs and Fokker–Planck, The multivariate Gaussian, EBMs, score matching, Langevin, Denoising diffusion (DDPM), Reverse SDEs, PF-ODE and samplers
    14. Flow matching, rectified flow and optimal transport · Needs: Score matching and diffusion, Flow matching and rectified flow, Positive definite matrices and Cholesky, Lagrange multipliers, KKT, duality, Joint distributions and covariance
    15. Flow matching on the sphere (Riemannian flow matching) · Needs: Flow matching and OT, Spherical harmonics and SFNO, Structured matrices in ML
    16. ML weather and climate models: architectures, training and evaluation · Needs: Chaos, DMD and SINDy, Data assimilation, Gaussian processes and UQ, Spherical harmonics and SFNO, Adjoints and neural ODEs, Score matching and diffusion, Graph neural networks, Transformers and ViT

    ML Systems

    1. GPU architecture: SMs, warps, the memory hierarchy and tensor cores · Needs: Matrices as maps, span and basis, Floating point and conditioning
    2. The roofline model and arithmetic intensity · Needs: GPU architecture, Matrices as maps, span and basis, Attention
    3. Kernels, tiling, coalescing and fusion · Needs: Roofline and arithmetic intensity, GPU architecture
    4. FlashAttention and online softmax · Needs: Kernels, tiling and fusion, Roofline and arithmetic intensity, Attention, Autodiff, LoRA, tensors, attention, Training a network
    5. Mixed precision: formats, loss scaling and accumulation · Needs: GPU architecture, Roofline and arithmetic intensity, Floating point and conditioning, Backpropagation, AdaGrad, RMSProp, Adam, AdamW
    6. Training memory: model states, activations and checkpointing · Needs: Mixed-precision training, FlashAttention and online softmax, Backpropagation, Transformers and ViT, AdaGrad, RMSProp, Adam, AdamW
    7. Collective communication and the alpha-beta cost model · Needs: GPU architecture, Matrices as maps, span and basis
    8. Data parallelism, ZeRO and FSDP · Needs: Collectives and the α-β model, Training memory and checkpointing, SGD and batch size, AdaGrad, RMSProp, Adam, AdamW
    9. Tensor and pipeline parallelism, and choosing a layout · Needs: Data parallelism, ZeRO and FSDP, Training memory and checkpointing, Collectives and the α-β model, Matrices as maps, span and basis
    10. Sequence, context and expert parallelism · Needs: Tensor and pipeline parallelism, FlashAttention and online softmax, Training memory and checkpointing, Collectives and the α-β model, Attention, Mixture of experts
    11. Quantization for training and inference · Needs: Roofline and arithmetic intensity, Mixed-precision training, Random variables and expectation
    12. LLM inference serving: KV cache, batching, paging and speculative decoding · Needs: Quantization, Roofline and arithmetic intensity, FlashAttention and online softmax, Training memory and checkpointing, Transformers and ViT, Attention, Random variables and expectation
    13. Profiling, MFU and compilation · Needs: Roofline and arithmetic intensity, Kernels, tiling and fusion, Training memory and checkpointing, Collectives and the α-β model, Data parallelism, ZeRO and FSDP

    MLOps

    1. Reproducible runs, versioning and data contracts · Needs: Python and NumPy for math, Common distributions
    2. Testing ML systems: unit, data, behavioural, slice and regression tests · Needs: Reproducible runs and data contracts, Common distributions, Hypothesis testing
    3. Containers, GPU images and CI pipelines · Needs: Testing ML systems, Reproducible runs and data contracts, GPU architecture
    4. Pipelines, Kubernetes and the cloud · Needs: Containers and CI, Common distributions, Derivatives in one variable
    5. Serving APIs, dynamic batching and queueing latency · Needs: Kubernetes and the cloud, Common distributions, Markov chains and MCMC
    6. Model registry, CD and safe deployment: shadow, canary, A/B · Needs: Serving and queueing, Reproducible runs and data contracts, Testing ML systems, Containers and CI, LLN, CLT and confidence intervals, Hypothesis testing, Bootstrap and A/B tests
    7. Distribution shift and drift statistics: PSI, KS and chi-square · Needs: Reproducible runs and data contracts, Jensen and KL divergence, Probability and Bayes' rule, Hypothesis testing
    8. Monitoring in production: alerts, delayed labels, feature stores and skew · Needs: Drift statistics, Serving and queueing, Registry and canary releases, Hypothesis testing
    9. LLMOps: versioning, eval gates, LLM judges and guardrails · Needs: Testing ML systems, Registry and canary releases, Hypothesis testing, Probability and Bayes' rule

    Time Series and Forecasting

    1. Time series data: trend, seasonality, noise and decomposition
    2. Stationarity, autocorrelation, ACF and PACF · Needs: Time series and decomposition
    3. Benchmark forecasts and honest evaluation: backtesting, metrics and leakage · Needs: Stationarity and autocorrelation, Time series and decomposition
    4. Exponential smoothing: from simple smoothing to Holt-Winters and ETS · Needs: Baselines and backtesting, Time series and decomposition, Loss surfaces and gradient descent
    5. Autoregressive and moving-average models (AR, MA, ARMA) · Needs: Stationarity and autocorrelation, Baselines and backtesting, Eigenvalues and Jordan form, QR and least squares, Structured matrices in ML
    6. ARIMA and SARIMA: differencing, forecasting and prediction intervals · Needs: AR, MA and ARMA, Stationarity and autocorrelation, Exponential smoothing, Baselines and backtesting
    7. State-space models and the Kalman filter · Needs: ARIMA and SARIMA, Exponential smoothing, AR, MA and ARMA, Positive definite matrices and Cholesky
    8. Forecasting with machine learning: lag features, gradient boosting and multi-step strategies · Needs: Baselines and backtesting, AR, MA and ARMA, Stationarity and autocorrelation
    9. Deep learning for forecasting: RNNs, temporal CNNs, transformers and foundation models · Needs: ML for forecasting, Baselines and backtesting, State space and Kalman filter, Convolutional networks, RNNs and LSTMs, Attention, Transformers and ViT, State-space models
    10. Probabilistic forecasting: quantiles, CRPS and conformal intervals · Needs: Baselines and backtesting, ARIMA and SARIMA, ML for forecasting
    11. Anomaly and change-point detection in time series · Needs: Probabilistic forecasting, Time series and decomposition, Baselines and backtesting, State space and Kalman filter, Drift statistics

    Causal Inference

    1. Correlation, causation and Simpson's paradox · Needs: Probability and Bayes' rule, Chance and data
    2. Potential outcomes, the fundamental problem and randomized experiments · Needs: Correlation vs causation, Random variables and expectation, LLN, CLT and confidence intervals, Hypothesis testing, Bootstrap and A/B tests
    3. Causal graphs: structural causal models, d-separation, confounders and colliders · Needs: Potential outcomes and experiments, Correlation vs causation, Joint distributions and covariance, The multivariate Gaussian
    4. Interventions and identification: the do-operator, backdoor and frontdoor adjustment, do-calculus · Needs: Causal graphs and d-separation, Potential outcomes and experiments, Joint distributions and covariance
    5. Adjustment, matching, propensity scores and inverse probability weighting · Needs: The do-operator and adjustment, Potential outcomes and experiments, Regression: the statistics view, Monte Carlo and importance sampling
    6. Instrumental variables: noncompliance, 2SLS and the local average treatment effect · Needs: Matching and propensity scores, Potential outcomes and experiments, The do-operator and adjustment, QR and least squares, Regression: the statistics view
    7. Regression discontinuity, difference-in-differences and synthetic control · Needs: Instrumental variables, Potential outcomes and experiments, Regression: the statistics view
    8. Heterogeneous effects and causal ML: meta-learners, double ML and causal forests · Needs: Potential outcomes and experiments, Matching and propensity scores, Inverse, determinant and projection
    9. Counterfactuals and mediation: questions about single cases and causal paths · Needs: Potential outcomes and experiments, Causal graphs and d-separation, The do-operator and adjustment, Bayesian inference
    10. Causal discovery: learning the graph from data · Needs: Causal graphs and d-separation, The do-operator and adjustment, Joint distributions and covariance, The multivariate Gaussian, Hypothesis testing
    11. Causality in ML practice: spurious features, distribution shift and off-policy evaluation · Needs: Correlation vs causation, Causal graphs and d-separation, Matching and propensity scores, Causal ML, Drift statistics, Bandits and exploration, Generalization

    ML Interview Prep

    1. Arrays and strings: hashing, two pointers, sliding window
    2. Stacks, binary search, heaps and greedy algorithms · Needs: Hashing, two pointers, windows
    3. Trees and graphs: traversals, BFS and DFS, topological sort, union-find, Dijkstra · Needs: Stacks, binary search, heaps, Hashing, two pointers, windows
    4. Backtracking and dynamic programming · Needs: Trees and graphs, Stacks, binary search, heaps
    5. Practical coding rounds: key-value store, LRU cache, rate limiter, crawler · Needs: Hashing, two pointers, windows, Trees and graphs
    6. ML coding in NumPy: softmax, logistic regression, k-means, AUC, MLP backprop · Needs: Backprop: MLPs, losses, layers, Backpropagation, Matrix calculus and matrix functions
    7. A training loop from scratch: AdamW, accumulation, mixed precision, BPE · Needs: ML coding in NumPy, AdaGrad, RMSProp, Adam, AdamW, Schedules, warmup and clipping, Backpropagation, Floating point and conditioning
    8. A transformer decoder from scratch: attention, RoPE, KV cache and decoding · Needs: Training loop, AdamW, BPE, Attention, Positional encodings, Normalization and residuals, Transformers and ViT, Autodiff, LoRA, tensors, attention
    9. Breadth: classic ML, evaluation and A/B testing · Needs: ML coding in NumPy, Registry and canary releases
    10. Breadth: deep learning, from initialization to diffusion and flow matching · Needs: Training loop, AdamW, BPE, Transformer decoder from scratch, Activations and initialization, Normalization and residuals, Training dynamics, AdaGrad, RMSProp, Adam, AdamW, Schedules, warmup and clipping
    11. Breadth: LLM pretraining, scaling laws, fine-tuning with LoRA, RLHF, DPO and evals · Needs: Training loop, AdamW, BPE, Transformer decoder from scratch, Jensen and KL divergence, Cross-entropy as a loss, Autodiff, LoRA, tensors, attention
    12. Training at scale: memory, parallelism, rooflines and infrastructure design · Needs: LLM breadth, Training loop, AdamW, BPE, Transformer decoder from scratch, Training memory and checkpointing, Collectives and the α-β model, Data parallelism, ZeRO and FSDP, Tensor and pipeline parallelism, Roofline and arithmetic intensity
    13. LLM serving: KV cache, batching, paged attention and quantization · Needs: Training at scale, Transformer decoder from scratch, Roofline and arithmetic intensity, Quantization, LLM inference serving, Serving and queueing
    14. ML system design: the framework, recommendation, search and ads · Needs: LLM serving, Classic ML and A/B tests, Training at scale, Self-supervised learning, Jensen and KL divergence, Registry and canary releases, Drift statistics, Monitoring and skew
    15. GenAI system design: RAG, assistants, agents and text-to-image · Needs: ML system design, LLM breadth, LLM serving, LLM inference serving, InfoNCE and contrastive learning, LLMOps and evals
    16. Behavioral interviews and the research talk · Needs: Training at scale, ML system design