Reproducible runs, versioning and data contracts

Before this lesson: nothing from this track; this is the first lesson. You need Python with NumPy and pandas (foundations 03), the idea of data leakage (from an introductory ML course) and the binomial distribution (probability 03). Time: about 70 minutes, plus the notebook.

---

Why this matters #

What MLOps is. MLOps (machine learning operations) is the practice of running ML models reliably in production: building them in a repeatable way, testing them, releasing them safely and watching them. Sculley et al. (NeurIPS 2015, "Hidden Technical Debt in Machine Learning Systems") drew a real ML system as many boxes: data collection, data verification, feature extraction, configuration, resource management, serving infrastructure, monitoring. The "ML code" is one small box in the middle. Most failures in production happen in the other boxes.

The work around a model is a loop, the ML lifecycle: data → validate and split → train → test → package → orchestrate → serve → release → monitor → back to data (retrain). Each lesson of this track covers one or two steps of the loop. The figure below shows which.

Two failures are common and silent, so we fix them first:

• A result cannot be rebuilt. Six months later "the same" training gives a different metric. A paper result, a bug report or an audit then has no solid ground.
• The input data changes under the model. An upstream team renames a column, changes a unit or adds a category. The model does not crash; it keeps producing numbers, now wrong ones.

Reproducibility makes every run a recorded function of known inputs. A data contract checks that every batch has the shape the model expects. Tests, CI and monitoring in the later lessons build on both.

Running example of the track. A rain-nowcast classifier for weather stations. Inputs: station id, time, humidity, temperature and pressure. Output: the probability of rain in the next hour. Some examples use seismic records (peak ground acceleration), a typical ML-for-science dataset.

1. Predict first #

2. The picture #

The ML system as a loop #

Interactive figure: loading…

A run is a function #

Think of a training run as a function of its inputs:

Reproducibility means: the same inputs give the same output. A tiny program such as np.random.default_rng(0).normal() has only three inputs (the code, the NumPy version and the seed), so it prints the same number every time. A training run has many hidden inputs: other random number generators, the order of floating-point additions, library versions, data files. Each input that you do not pin or record is a way for two runs to differ.

There are two levels:

• Bitwise reproducibility: the same bits in every weight. It is possible only with the same hardware, software and settings. You want it for debugging and for regression tests.
• Statistical reproducibility: the same metrics within the run-to-run spread, measured over several seeds. A paper or a model comparison needs it.

The smallest example of nondeterminism. Floating-point addition is not associative. In float64,

Each addition rounds its result to the nearest representable number, so the result depends on the order of the additions (see linear algebra 13 for floating-point arithmetic). A GPU adds many numbers in parallel, and the order can change from run to run. Training then amplifies these tiny differences over thousands of steps.

Interactive figure: loading…

A stable split #

To compare models over months, the test set must stay fixed while new data arrives. A seeded random split does not do this: it shuffles all rows, so its result depends on the number of rows. A hash split gives each row a number $u(k) \in [0, 1)$ computed only from the row's key $k$ (for example the station id), and puts the row into a split by comparing $u(k)$ with fixed boundaries. New rows cannot change the number of an old row.

Interactive figure: loading…

A data contract is a type signature for a table #

A type signature says "this argument is a float". A data contract says "column pga_g is a float, at most 10% of its values are missing, and every value is between 0 and 5". Each check looks only at the current batch. That is its strength (it is cheap, exact and easy to explain) and its limit (it never compares the batch with the training data).

Interactive figure: loading…

3. Definitions and main results #

Sources of nondeterminism and their controls #

The list uses PyTorch as the example. JAX and TensorFlow have the same sources and similar settings.

Each item gives the source, what changes between runs, and the control.

• Random number generators (Python random, NumPy, PyTorch). Changes: initial weights, dropout masks, data augmentation. Control: seed all of them; torch.manual_seed seeds the CPU and every CUDA device.
• Data-loader shuffling. Changes: the order of the samples. Control: a seeded generator for the sampler; in distributed training also a seeded sampler, with the epoch number set every epoch.
• Data-loader workers (separate processes). Changes: random augmentations inside each worker. Control: each worker gets its own seed from the main process; seed any extra generator from it (see the trap below).
• cuDNN autotuning. Changes: which convolution algorithm runs (the algorithms add in different orders). Control: torch.backends.cudnn.benchmark = False and torch.backends.cudnn.deterministic = True.
• GPU kernels with atomic adds. Changes: the order of the additions. Control: in PyTorch, use_deterministic_algorithms(True): deterministic versions where they exist, an error where none exists.
• Reduction order in collectives (the all-reduce over GPUs). Changes: the order of the sum over GPUs. Control: the same number of GPUs, the same topology and the same communication settings.
• Library, driver and hardware versions. Changes: which kernels run and their order of additions. Control: a lockfile or a container image digest; record the GPU type and the driver.

An atomic add is a GPU operation that lets many threads add into the same memory cell without losing updates; the threads finish in an unpredictable order. An all-reduce sums a tensor over all GPUs and gives every GPU the result (see ML systems 07).

A typical setup for bitwise reproducibility in PyTorch (an illustration; it needs PyTorch and a GPU):

import os, random
import numpy as np
import torch

os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"   # cuBLAS needs this for deterministic GEMMs
random.seed(0); np.random.seed(0); torch.manual_seed(0)
torch.backends.cudnn.benchmark = False             # no autotuning
torch.backends.cudnn.deterministic = True
torch.use_deterministic_algorithms(True)           # error on ops without a deterministic version
g = torch.Generator().manual_seed(0)               # pass as DataLoader(..., generator=g)

The PyTorch documentation states the limit: "Completely reproducible results are not guaranteed across PyTorch releases, individual commits, or different platforms." Deterministic settings also make some operations slower.

The worker trap. A data loader starts several worker processes, and each worker receives a copy of the dataset object. If the dataset stores a generator object (for example self.rng = np.random.default_rng(0) in its constructor), every worker gets a copy with the same state, so all workers draw the same "random" numbers. PyTorch reseeds its own generator, Python's random and NumPy's global generator in each worker, but it cannot know about a generator object inside your dataset. The fix is one independent stream per worker. In NumPy, np.random.SeedSequence(seed).spawn(n) makes $n$ child seeds whose streams do not overlap:

import copy
import numpy as np

rng = np.random.default_rng(0)
workers = [copy.deepcopy(rng) for _ in range(2)]        # what a copy into 2 workers does
print([round(float(w.random()), 4) for w in workers])   # [0.637, 0.637]   the same numbers

streams = [np.random.default_rng(s) for s in np.random.SeedSequence(0).spawn(2)]
print([round(float(g.random()), 4) for g in streams])   # [0.9429, 0.6772] independent

Versioning: four things in one record #

A run is reproducible only if every input has a name that cannot move. Record four versions:

1. Code: the git commit SHA, plus a "dirty" flag (or the diff) if the working tree had uncommitted changes.
2. Environment: the lockfile (the exact version of every package) or, better, the container image digest (sha256:..., lesson 03). A tag such as latest is not a version: it can point to a different image tomorrow.
3. Data: a content hash of the exact files or rows, plus the split rule and its salt. A path such as /data/stations/2024/ is not a version: the files behind it can change. Tools such as DVC store each data file under its hash and put a small pointer file (with the hash) into git.
4. Model: the checkpoint's location and content hash, linked to the run that produced it.

A content hash (here SHA-256) is a short fingerprint computed from the bytes: if one byte changes, the hash changes; if nothing changes, the hash is the same on every machine.

An experiment tracker (such as MLflow or Weights & Biases) keeps one record per run. To reproduce a run, the record needs:

• the four versions above;
• the full resolved config (after defaults and command-line overrides are applied), and its config hash (SHA-256 of the config written as canonical JSON, with sorted keys);
• the seeds and the determinism settings;
• the hardware and the launch: GPU type and count, number of processes, the job id.

To compare runs, it also needs the metrics at each step, the final metrics per split and per data slice, and the artifacts (checkpoints, plots).

Theorem (hash-based splits are stable) #

Setting. Each row has a key $k$: a sample id, or a group id such as a station, a patient or a storm. Let $s$ be a salt (a fixed string that you choose, such as split-v1) and $B$ a large integer (here $B = 10^6$). Let $h(x)$ be the first 8 bytes of the SHA-256 hash of the string $x$, read as an unsigned integer. Define

where $\|$ joins strings (so the hashed string is split-v1:ZUR). For split fractions $f_1, \dots, f_m$ that sum to 1, let $c_i = f_1 + \dots + f_i$ be the running sums. Key $k$ goes to the first split $i$ with $u(k) < c_i$.

Claims. (a) The split of a key never changes when rows are added, removed or reordered. (b) All rows with the same key are in the same split. (c) With many keys, split $i$ receives a fraction close to $f_i$. (d) A different salt gives an independent split that agrees with the old one on a fraction $\sum_i f_i^2$ of the keys.

Why this is true, in one sentence. The number $u(k)$ is a function of the key alone, and the boundaries are fixed; nothing else enters.

Proof.

1. $u(k)$ is computed from $s$ and $k$ only. No other row, no row count and no row order enters the formula.
2. The split depends only on $u(k)$ and the fixed running sums $c_i$. With step 1, this gives (a).
3. Rows with the same key have the same $u$, so they get the same split. This gives (b): no group can be in two splits, so there is no leakage of a group between splits.
4. SHA-256 behaves like a uniform random function, so $u(k)$ is close to uniform on $[0, 1)$ (up to steps of $1/B$). Then $P(\text{split } i) = c_i - c_{i-1} = f_i$, with $c_0 = 0$. Out of $n$ distinct keys, the number in split $i$ has the binomial distribution $\mathrm{Bin}(n, f_i)$, so its fraction has standard deviation $\sqrt{f_i(1 - f_i)/n}$. This gives (c).
5. Under two salts $s_1, s_2$, the numbers $u_1(k)$ and $u_2(k)$ behave like independent uniform numbers. Write $S_1, S_2$ for the split of a key under each salt. The key keeps its split if both numbers fall into the same split: $P(S_1 = S_2) = \sum_i P(S_1 = i)\,P(S_2 = i) = \sum_i f_i^2$. For $(0.8, 0.1, 0.1)$ this is $0.64 + 0.01 + 0.01 = 0.66$. This gives (d). $\blacksquare$

A picture for step 5: put $u_1$ on the horizontal axis and $u_2$ on the vertical axis of the unit square. The points are spread uniformly. A key keeps its split when its point lies in one of the squares on the diagonal, of sides $f_1, f_2, \dots$, so the probability is the total area $\sum_i f_i^2$.

Why not Python's hash()? For a string, hash() is salted with a random value in every new Python process (unless the environment variable PYTHONHASHSEED is fixed). Step 1 fails: the same key gets a different number tomorrow.

Why not a seeded random split? train_test_split(ids, test_size=0.1, random_state=0) draws a random permutation of all $n$ rows and takes the first 10% as test. The permutation depends on $n$, so step 1 fails: one added row reshuffles everything.

Group leakage and time leakage. A hash split by station prevents leakage between stations (no station is in train and test). It does not prevent leakage in time: training and test stations cover the same days, and weather is correlated in space, so the model has already seen those days at nearby stations. For forecasting, also split by time: train on the past, test on the future.

Data contracts #

A data contract is a machine-checkable agreement between the producer of a dataset and its consumer. For each column it states:

• presence: the column is required or optional;
• dtype: int, float, bool, string or datetime;
• nullability: whether missing values are allowed, and the maximum fraction of nulls;
• range: the inclusive minimum and maximum;
• domain: the allowed values of a categorical column;
• uniqueness: for keys, no value appears twice.

Each check returns the number of offending rows. What happens next depends on where the check runs:

Where	Typical checks	On failure
ingestion (data enters your system)	the full contract on each batch	block the batch, or quarantine it (store it aside for inspection) and keep using the last good data
before training	the full contract on the training snapshot	fail the pipeline: never train on data that fails
serving (each request)	required fields, dtypes, ranges	reject the request (HTTP status 422) or use a documented fallback
monitoring (over time)	null rates; distributions against a reference	alert a human; do not block

Hard, unambiguous violations block. Statistical signals alert. Silently dropping bad rows is the worst option: it hides the problem, and it changes the data distribution without anyone knowing.

Result (what schema checks can and cannot see) #

A schema check is a check of a batch against the contract's rules (presence, dtype, nulls, range, domain, uniqueness).

They catch: a renamed or dropped column; a type change, including the classic one (one missing value turns a pandas int64 column into float64, because NaN is a float); a new category ("hnz" instead of "HNZ"); and a unit change if it moves values out of range (Kelvin values in a Celsius column with range $[-60, 60]$).

They miss: any change of the distribution that stays inside the allowed values.

Proof.

1. A schema check on a column is a function of the current batch only: "every non-null value lies in a set $S$", plus counts of nulls and duplicates.
2. Let $P$ be the training distribution and $Q$ the new one. If both put all their mass inside $S$ (with the same null and duplicate behaviour), a batch from $Q$ passes exactly like a batch from $P$.
3. The check uses no reference sample, so it has no way to tell $P$ from $Q$. $\blacksquare$

Examples: wind in km/h instead of m/s that stays below the maximum; a stuck sensor that repeats a valid value; an integer feature that a code bug sets to $-1$ on one slice of the traffic. The last one is example 1.1 of Breck et al. (2019): $-1$ was a valid value, so the data "looked perfectly fine" to the checks. These failures need checks against a reference distribution (PSI and the KS test, lesson 07), run-length checks or volume checks.

Answers to section 1. (1) Not guaranteed. By default cudnn.deterministic is False, so cuDNN may choose nondeterministic algorithms (and many scripts also turn on cudnn.benchmark); some CUDA kernels use atomic adds. Both change the order of floating-point additions, so the bits can differ. You need the determinism settings of the list and the same hardware and software. (2) About 80% (80.3% with scikit-learn 1.9). The new permutation of 12,000 rows has nothing to do with the old one, so an old test row lands in the new training set with probability about 0.8. Even one added row moves about 58% of the old test rows into training. Any before/after comparison on the old test set is then contaminated: the new model was trained on most of it.

4. Worked example #

(a) A hash split by hand. Salt split-v1, fractions train 0.8, val 0.1, test 0.1, so the boundaries are $c_1 = 0.8$, $c_2 = 0.9$, $c_3 = 1$. The key is the station code:

key	hashed string	$u$	rule	split
ZUR	split-v1:ZUR	0.897907	$0.8 \le u < 0.9$	val
BAS	split-v1:BAS	0.676145	$u < 0.8$	train
SIO	split-v1:SIO	0.309680	$u < 0.8$	train
LUG	split-v1:LUG	0.950265	$0.9 \le u$	test

Every record of station ZUR, past and future, is in validation. Add 50 new stations next year: these four do not move. With salt split-v2, ZUR has $u = 0.821732$ (val again) and BAS has $u = 0.241780$ (train again). That is chance: each key keeps its split with probability 0.66.

(b) A contract for ground-motion records. Peak ground acceleration (PGA) is the largest acceleration of the ground during an earthquake, here in units of $g = 9.80665$ m/s². The contract:

• record_id: int, unique;
• station: str, not nullable;
• component: str, allowed {HNE, HNN, HNZ};
• magnitude: float, not nullable, range $[0, 10]$;
• pga_g: float, nullable with at most 10% nulls, range $[0, 5]$.

record_id	station	component	magnitude	pga_g
101	ZUR	HNE	3.1	0.012
102	BAS	HNN	4.2	NaN
103	null	HNZ	2.8	0.004
104	SIO	HHZ	5.0	0.31
105	ZUR	HNE	3.3	9.81
105	BAS	HNN	3.3	0.002
107	SIO	hnz	12.0	NaN
108	ZUR	HNZ	2.9	0.020

Go through the contract column by column:

1. record_id unique: 105 appears twice, so 2 rows violate.
2. station not nullable: 1 row (record 103).
3. component allowed: 2 rows. HHZ is a different channel type; hnz has the wrong case.
4. magnitude range $[0, 10]$: 1 row (12.0).
5. pga_g null fraction: 2 nulls in 8 rows, $2/8 = 25\% > 10\%$ (2 rows).
6. pga_g range $[0, 5]$: 1 row (9.81, almost surely a value in m/s², not in $g$).

That is six violation groups. The two NaNs are not range violations: nulls are counted only by the null checks. (Otherwise one missing value would be reported twice.)

(c) The same contract misses a unit change. Suppose the producer switches all of pga_g from $g$ to m/s² (multiply by 9.80665). The valid values in the table become

$0.012 \to 0.118$, $0.004 \to 0.039$, $0.31 \to 3.040$, $0.002 \to 0.020$ and $0.020 \to 0.196$.

All of them are still inside $[0, 5]$. Only records above $5 / 9.80665 = 0.51\,g$, which are rare strong motions, would fail. So the answer to predict-first question 3 is: almost none. The contract passes almost every batch while every value is wrong by a factor 9.8. On a log scale this is a shift of $\log_{10} 9.80665 = 0.99$. A check of the $\log_{10}$ PGA distribution against a reference batch catches it at once (the last figure of section 2).

(d) The int-to-float trap. If one record_id is missing, pandas stores the column as float64, because NaN is a float. The contract then reports two violations: the dtype (float, not int) and the null. Fix it upstream, or read the column with pandas' nullable integer dtype Int64, which can hold a missing value and stays an integer type.

5. In code #

These checks need only NumPy, pandas, scikit-learn and the standard library. The notebook has them and the exercises.

import hashlib
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

# 1. Floating-point addition is not associative
print((0.1 + 0.2) + 0.3 == 0.1 + (0.2 + 0.3))          # False

# 2. The order of a float32 sum changes the last digits
x = np.random.default_rng(1).random(1000).astype(np.float32)
s_fwd = np.float32(0)
for v in x:
    s_fwd = np.float32(s_fwd + v)
s_bwd = np.float32(0)
for v in x[::-1]:
    s_bwd = np.float32(s_bwd + v)
print(s_fwd, s_bwd)                                     # 502.80438 502.80484

# 3. The hash split rule of the theorem
def u(key, salt="split-v1", B=10**6):
    h = int.from_bytes(hashlib.sha256(f"{salt}:{key}".encode()).digest()[:8], "big")
    return (h % B) / B
print([u(k) for k in ["ZUR", "BAS", "SIO", "LUG"]])    # [0.897907, 0.676145, 0.30968, 0.950265]

# 4. A seeded random split moves old rows when the data grows
_, test_old = train_test_split(np.arange(10_000), test_size=0.1, random_state=0)
train_new, _ = train_test_split(np.arange(12_000), test_size=0.1, random_state=0)
print(np.isin(test_old, train_new).mean())              # 0.803 (scikit-learn 1.9)

# 5. One missing value turns an int column into float64
print(pd.Series([101, 102, None, 104]).dtype)                   # float64
print(pd.Series([101, 102, None, 104], dtype="Int64").dtype)    # Int64

np.sum of the same array uses pairwise summation, a third order of additions; it gives 502.80463. None of the three is "wrong": they are three roundings of the same exact sum.

6. Where it shows up #

• Model comparison and papers. A gain of 0.3 points means little without the seed-to-seed spread. Report the mean and the spread over several seeds (statistical reproducibility).
• Science and HPC. Many climate and weather codes test that a restarted run is bit-for-bit identical to an uninterrupted one. An ML emulator inside such a workflow inherits the requirement. On a cluster, record the job id, the node type and the image digest.
• Unit changes in scientific data. These are the classic contract failures. ERA5 reanalysis stores geopotential in m² s⁻² (divide by $g$ to get a height in metres) and total precipitation in metres of water. Seismic waveforms arrive in raw instrument counts or in physical units, depending on whether the instrument response was removed.
• Industry. A fraud model that splits by transaction id, not by customer, leaks customers between train and test and looks better offline than online. A hash split by customer id fixes it. Feature pipelines at large companies run schema checks on every batch before training (Breck et al. 2019 describe the system at Google).

7. Common mistakes #

• Seeding only one generator. Seed Python, NumPy and the framework, and do not store one generator object in a dataset that is copied into workers.
• Turning on cudnn.benchmark for speed, then being surprised that two runs differ.
• A seeded random split on growing data, or a row-level split when rows of one station, patient or storm are correlated (group leakage).
• Logging the config file instead of the resolved config, a data path instead of a hash, or an image tag instead of a digest.
• Reading a passing schema check as "the data is fine". It only says that the values are allowed, not that they come from the same distribution.
• Dropping invalid rows silently instead of counting them, quarantining the batch and alerting.

8. Check yourself #

1. Fractions are train 0.7, val 0.15, test 0.15. You change the salt. What fraction of keys keep their split?
   Answer

   $0.7^2 + 0.15^2 + 0.15^2 = 0.49 + 0.0225 + 0.0225 = 0.535$.
2. A temperature contract has range $[-60, 60]$ (°C). Upstream switches to °F. Which true temperatures does the range check catch?
   Answer

   Only readings with $F > 60$ or $F < -60$. With $F = 9C/5 + 32$, that is $C > 15.6$ °C or $C < -51.1$ °C. A batch from a cool day passes completely, so you also need a distribution check.
3. Your 4-GPU training run is bitwise reproducible. You move to 8 GPUs with the same seed and settings. What can you promise, and why?
   Answer

   Only statistical reproducibility. The batches per GPU and the order of the all-reduce sum change (often the effective batch size too), so the floating-point results change. Compare metrics over several seeds.
4. Why does a hash split by station prevent leakage between stations but not leakage in time?
   Answer

   All rows of one station share its key, so they share a split. But training and validation stations cover the same dates, and weather is correlated in space, so the model has already seen those days at nearby stations. For forecasting, also split by time.
5. Which of these does a schema check catch? (a) a renamed column; (b) wind switched from m/s to km/h, with maximum 200; (c) a new station code in an allowed-values column; (d) a sensor stuck at 12.0 °C.
   Answer

   (a) Yes: a missing column. (b) Mostly no: values grow by a factor 3.6 but stay below 200 unless the wind is above 55.6 m/s. (c) Yes: a domain violation. (d) No: 12.0 is a valid value; you need a run-length or variance check.
6. Two configs are {"lr": 0.001, "batch": 64} and {"batch": 64, "lr": 0.001}. Should their config hashes be equal? How do you make sure they are?
   Answer

   Yes: they are the same configuration. Write the config as canonical JSON (sorted keys, fixed separators) before hashing, so that the key order does not change the bytes.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. What is the difference between bitwise and statistical reproducibility?

   Show answer

2. Why can two GPU runs with the same seed give different weights?

   Show answer

3. Compute $(0.1 + 0.2) + 0.3$ and $0.1 + (0.2 + 0.3)$ in float64.

   Show answer

4. Why is an image tag such as latest not a version?

   Show answer

5. What is the rule of a hash split?

   Show answer

6. Why not use Python's built-in hash() for a split?

   Show answer

7. Why does a seeded random split move old rows when data is added?

   Show answer

8. Fractions 0.7/0.15/0.15, new salt: what fraction of keys keep their split?

   Show answer

9. Why can a schema check not detect a unit change that stays inside the allowed range?

   Show answer

10. What happens to a pandas int64 column when one value is missing?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "How do you make a training run reproducible?" — Pin and log the code (commit SHA), the environment (lockfile or image digest), the data (content hash), the config (resolved and hashed), the seeds, the determinism settings and the hardware. Then say what still varies when the number of GPUs changes.
• "A colleague cannot reproduce your result on another cluster. What do you check?" — The logged versions first (image, data hash, config), then GPU type, driver and library versions, number of processes and determinism settings. Then measure the seed-to-seed spread before you call it a bug.
• "How do you split data for a model that will be retrained every month?" — A hash split on the right group key (station, user, patient) with a fixed salt, plus a time split for forecasting.
• "Where do you validate data, and what happens on failure?" — At ingestion (block or quarantine), before training (fail the pipeline), at serving (reject with 422 or a fallback), in monitoring (alert). Never drop rows silently.

10. Practice #

The notebook for this lesson runs in Colab. It has five exercises, each with a check cell and a folded solution:

1. The worker trap: see two copied generators draw the same numbers, then write make_streams(seed, n_workers) with SeedSequence.spawn.
2. Write hash_bucket(key, n_buckets, salt) and assign_splits(keys, fractions, salt); test that adding keys moves no old key and that the fractions are within $3\sqrt{f(1-f)/n}$.
3. Write config_hash(cfg): equal for reordered dicts, different for a changed value.
4. Write validate(df, contract) and find exactly the six violation groups of worked example (b).
5. Show that the contract passes a unit-changed batch while a comparison of $\log_{10}$ quantiles with a reference batch flags it.

11. Watch and read #

Lecture 04: Data Management (FSDL 2022)The Full Stack · Full Stack Deep Learning 2022 · 35 min

Data versioning, storage and labelling in a real ML team; compare with the four versions of section 3.

Mateusz Opala: Reproducible Machine Learning | PyData Warsaw 2019PyData · 30 min

Why ML results are hard to rebuild and which inputs to pin.

Lab 04: Experiment Management (FSDL 2022)The Full Stack · Full Stack Deep Learning 2022 · 32 min

An experiment tracker in practice: what a run record contains and how runs are compared.

• Sculley et al., Hidden Technical Debt in Machine Learning Systems (NeurIPS 2015): figure 1 is the "small ML code box" picture of section "Why this matters".
• PyTorch documentation, Reproducibility: controlling sources of randomness, deterministic algorithms, the data loader.
• NumPy documentation, Parallel random number generation: SeedSequence.spawn for independent streams.
• Made With ML, Versioning and Experiment tracking.
• Breck et al., Data Validation for Machine Learning (SysML 2019): example 1.1, section 3 (single-batch validation) and section 4 (inter-batch validation).