Videos and books
The free courses, lecture series, books and short reads that pair best with the MLOps track, with a note on what each one is good for.
Every resource on this page is free to watch or read, from the author's or the publisher's own page. Where a standard book is not free, we say so and point to its official page. Each lesson of the track also ends with a Watch and read section that names the exact lecture, page or paper.
How to use it. MLOps is learned by doing. Watch one course for the whole picture, read the official documentation of each tool you use, and read the few papers that the lessons build on. A good default: one lecture before a lesson, then the lesson and its notebook, then the reading from the table at the end of this page. Tools change quickly; the ideas of this track (versions, tests, queues, gates, drift statistics) do not.
Video courses #
Whole-course overviews #
From 47:24, the lifecycle of an ML project; compare it with the loop of lesson 01.
- Full Stack Deep Learning 2022 (course page with notes, YouTube playlist). Lectures and labs on the whole path from data to a deployed model: data management, experiment management, testing, deployment and monitoring. The lessons of this track link single lectures of this course. Pairs with lessons 01, 02, 05 and 08.
- MLOps Zoomcamp (DataTalksClub; course repository, YouTube playlist). A free hands-on course with homework: experiment tracking, orchestration, deployment, monitoring and best practices (tests, CI). Pairs with lessons 01, 03, 04 and 08.
- Made With ML (Goku Mohandas; course). A written course with code from a notebook to a tested, versioned and monitored service. Its pages on testing, versioning, CI/CD and monitoring are linked from lessons 01, 02, 03 and 08.
What MLOps means and the steps of the course project; a short start before lesson 01.
ML systems design #
ML in research versus in production, and an iterative design process from scoping to maintenance.
- Stanford CS 329S, Machine Learning Systems Design (Chip Huyen; course page with notes and slides). Data engineering, deployment, distribution shift and monitoring, with guest lectures from industry. Pairs with lessons 05, 07 and 08.
Why the data, and not only the model, must be versioned, checked and improved; the idea behind data contracts.
What to log and monitor in production, and how data drift shows up in real systems.
Tools: containers and Kubernetes #
Images, containers, registries and a Dockerfile, step by step; watch it before lesson 03 if Docker is new to you.
Pods, Deployments, Services and YAML configuration with a running example; the first hour is enough for lesson 04.
Free books #
- Vijay Janapa Reddi et al., Machine Learning Systems (free online textbook). An open textbook on the engineering of ML systems, from data engineering to deployment. Its chapter on ML operations pairs with lessons 03–06 and 08.
- Google, Site Reliability Engineering (free online) and The Site Reliability Workbook (free online). The standard books on running services: SLOs, monitoring, alerting and safe releases. Read "Monitoring Distributed Systems" (lesson 08), "Alerting on SLOs" (lessons 05 and 08) and "Canarying Releases" (lesson 06).
- Stas Bekman, Machine Learning Engineering Open Book (GitHub). Practical notes on training and serving large models: hardware, storage, networking, debugging and reproducibility. Pairs with lessons 01, 03 and 04.
- Zhang, Lipton, Li and Smola, Dive into Deep Learning, section Environment and Distribution Shift. Covariate, label and concept shift and their corrections, with proofs. Pairs with lesson 07.
- Not free, but standard: Chip Huyen, Designing Machine Learning Systems (O'Reilly, 2022; author's page); Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments (Cambridge, 2020; book page) for A/B tests and canaries (lesson 06); Mor Harchol-Balter, Performance Modeling and Design of Computer Systems (Cambridge, 2013; book page) for the queueing theory of lesson 05.
Short reads #
- Sculley et al., Hidden Technical Debt in Machine Learning Systems (NeurIPS 2015). Why the ML code is a small part of an ML system. Read it first.
- Google, Rules of Machine Learning. 43 short rules from practice; the rules on training-serving skew (#29–#37) are the basis of lesson 08.
- Google, Production ML systems (Machine Learning Crash Course). Static versus dynamic training and inference, data dependencies and pipeline tests, in short pages.
- Breck et al., The ML Test Score (IEEE Big Data 2017). The 28 tests of lesson 02.
- Google Cloud, MLOps: Continuous delivery and automation pipelines in machine learning. The maturity levels 0, 1 and 2 and continuous training (lessons 03 and 06).
- Sato, Wider and Windheuser, Continuous Delivery for Machine Learning (martinfowler.com, 2019). Versioning of code, data and models together, and a deployment pipeline for models (lessons 01, 03 and 06).
- Chip Huyen, MLOps guide (a reading list) and Data distribution shifts and monitoring (lessons 07 and 08).
- Shankar, Garcia, Hellerstein and Parameswaran, Operationalizing Machine Learning: An Interview Study (2022). What 18 ML engineers say about experiments, deployment and monitoring.
- Paleyes, Urma and Lawrence, Challenges in Deploying Machine Learning: a Survey of Case Studies (ACM Computing Surveys, 2022). Problems at each step of the lifecycle, from published case studies.
- J. Dean and L. A. Barroso, "The Tail at Scale" (CACM 2013). Tail latency and fan-out (lesson 05).
- Hamel Husain, Your AI product needs evals and the LLM evals FAQ; Eugene Yan, Patterns for building LLM-based systems and products (lesson 09).
Which lesson pairs with what #
| Lesson | Watch | Read |
|---|---|---|
| 01 Reproducible runs and data contracts | FSDL 2022, Lecture 04 (Data Management) and Lab 04 (Experiment Management) | Sculley et al. 2015; Breck et al., Data Validation for ML (SysML 2019); PyTorch notes on reproducibility |
| 02 Testing ML systems | FSDL 2022, Lecture 03 (Troubleshooting & Testing); Ribeiro's CheckList talk | Ribeiro et al. 2020 (CheckList), section 2; Breck et al. 2017 (ML Test Score) |
| 03 Containers and CI | TechWorld with Nana, Docker crash course | Docker docs on the build cache and multi-stage builds; NVIDIA CUDA Compatibility |
| 04 Kubernetes and the cloud | TechWorld with Nana, Kubernetes course (first hour) | Kubernetes docs: Pods, probes, HPA; Airflow DAG Runs; Young 1974 |
| 05 Serving and queueing | FSDL 2022, Lecture 05 (Deployment); MIT 6.02 on Little's law | Sigman, Notes on Little's Law; Dean and Barroso 2013; Triton Batchers |
| 06 Registry and canary releases | StatQuest on power analysis; Johari et al. on peeking | SRE Workbook, Canarying Releases; Google Cloud MLOps levels; Johari et al. (always-valid p-values) |
| 07 Drift statistics | Smola, CS329P lecture on covariate shift | Dive into Deep Learning, distribution shift; Rabanser et al. 2019; Lipton et al. 2018 |
| 08 Monitoring and skew | FSDL 2022, Lecture 06 (Continual Learning); CS 329S monitoring lecture | Rules of ML #29–#37; Feast Point-in-time joins; SRE book, Monitoring Distributed Systems |
| 09 LLMOps and evals | Hamel Husain, AI Eval Concepts; JJ Allaire on Inspect | Husain, Your AI product needs evals; Zheng et al. 2023 (LLM-as-a-judge); OWASP LLM01 |