Pipelines, Kubernetes and the cloud

Before this lesson: lesson 03 (images and digests). For section 3.8: the exponential distribution and the Poisson process (probability lesson 03), and the minimum of a function of one variable (calculus lesson 01). Time: about 70 minutes, plus the notebook.

---

Why this matters #

Almost every ML system has a pipeline that runs on a schedule (ingest new data, compute features, retrain, evaluate, publish forecasts) and a service that answers requests. Pipelines run in schedulers such as Apache Airflow, Prefect, Dagster or Argo Workflows. Services run on Kubernetes, directly or through tools built on it (KServe, Ray Serve). Datasets, checkpoints and model files live in object storage (Amazon S3, Google Cloud Storage, Azure Blob Storage). Training often uses spot GPUs, which cost a fraction of the normal price but can be taken back.

The failures here are quiet and expensive: a retried task that writes its rows twice, a server that restarts forever while it loads a large model, a checkpoint that was half written, a bill for data transfer that is larger than the bill for GPUs. This lesson gives you the rules and the small formulas that prevent them. Lesson 05 builds the model server that runs inside these Pods.

1. Predict first #

2. The picture #

Three ideas carry this lesson.

Idea	Meaning	Where you meet it
Reconciliation	You declare the desired state; a control loop compares it with the observed state and acts on the difference, forever.	Kubernetes Deployments, autoscalers
Immutability	An image or a stored object never changes after it is written. A new version is a new object with a new name or digest.	container images, object storage, datasets
Idempotency	Running a task twice with the same inputs leaves the same state as running it once.	scheduled tasks, retries, backfills

Idempotency, as a picture. A daily task writes the rows of one day as several files ("parts") under one folder ("prefix") of a bucket. Attempt 1 crashes after a few parts, and the scheduler retries. What the next task reads depends on how the task writes.

Interactive figure: loading…

Reconciliation and probes, as a picture. A Kubernetes Deployment declares "keep 3 copies of this server". Each copy runs in a Pod. If a Pod dies, the control loop sees 2 observed Pods instead of 3 desired ones and creates a new Pod. Three probes tell Kubernetes about each container: is it alive (liveness), may it receive traffic (readiness), has it finished starting (startup)? A model that needs 90 seconds to load makes the difference between them visible.

Interactive figure: loading…

3. Definitions and main results #

3.1 DAG schedulers and idempotent tasks #

A DAG (directed acyclic graph) is a set of tasks with dependencies, for example ingest → check data → features → train → evaluate. "Acyclic" means that no task depends on itself through a chain, so there is always an order in which to run them. A DAG run is one execution of the whole graph. Apache Airflow is the most common scheduler; Prefect, Dagster and Argo Workflows have the same ideas with other names.

• The schedule says when runs are created (for example @daily, or a cron expression such as 0 6 * * *, "every day at 06:00").
• The data interval $[\text{start}, \text{end})$ is the time range a run processes. The logical date of the run is the start of its data interval.
• Retries rerun a failed task, usually after a delay.
• A backfill creates runs for a past range on purpose, for example after a bug fix.
• Catchup creates the runs that were missed since the start date when a DAG is switched on.

A task is idempotent if running it twice with the same inputs leaves the same state as running it once. Every scheduled task must be idempotent, because retries, backfills and manual reruns all run a task again, often on top of a half-written output. The recipe:

1. Parameterise the task by the run's logical date or data interval, never by now(). Otherwise a rerun next week processes next week's data.
2. Replace, do not append. Write to an output keyed by the interval (.../date=2026-01-03/) and replace it as a whole, or upsert rows by a key.
3. Commit atomically. Write the parts under a prefix unique to the attempt, then write a manifest (or a _SUCCESS marker) last. Readers trust only outputs named in a manifest.
4. Keep no hidden state between runs (counters in memory, files on the local disk), and fix the seeds.

Step 3 works on object storage because one write of one object is atomic: a reader sees the old manifest or the new one, never half of it (section 3.6).

3.2 Kubernetes objects #

Kubernetes runs containers on a cluster of machines (nodes). You describe objects in YAML; controllers make the cluster match them.

Object	What it is	ML use
Pod	one or more containers scheduled together on one node, sharing network and volumes	one serving replica; one training worker
Deployment	keeps $N$ identical Pods running; replaces them one by one on an update (rolling update) and can roll back	a model server
Service	a stable name and virtual IP address that spreads requests over the ready Pods with a given label	the model's endpoint
Job	runs Pods until they finish successfully, and retries failures	batch inference, evaluation, a training run
CronJob	creates a Job on a cron schedule	a nightly drift report

Requests and limits. Each container states resources:

• requests are used for scheduling: the scheduler puts a Pod only on a node where the sum of the requests fits;
• limits are enforced while the container runs: CPU use above the limit is throttled (slowed down); memory use above the limit gets the container OOM-killed (out of memory).

The autoscaler of section 3.4 measures CPU utilisation as a percentage of the request. If a container has no CPU request, the Pod has no CPU utilisation and the autoscaler does nothing for that metric.

GPUs. A vendor device plugin advertises GPUs to Kubernetes as a resource such as nvidia.com/gpu. You set it under limits (a request, if given, must equal the limit). A GPU is given to one container as a whole device; sharing one GPU between containers needs extra setup (time-slicing, or MIG partitions on some data-center GPUs).

3.3 Liveness, readiness and startup probes #

Probe	Question	When it fails, Kubernetes …	Check it with
liveness	Is the process alive (not stuck)?	restarts the container	a cheap endpoint such as /healthz that never depends on the model or a database
readiness	May this Pod get traffic now?	removes the Pod from the Service, without a restart	/readyz: model loaded and not overloaded
startup	Has the container finished starting?	restarts the container if it does not succeed within its budget	/readyz; budget = failureThreshold × periodSeconds

While a startup probe has not yet succeeded, the liveness and readiness probes do not run. So a slow model load needs a startup probe whose budget is larger than the worst load time. A restarted container is restarted with a growing back-off delay, so a restart loop also wastes more and more time.

A Deployment for the rain-nowcast model server, as an illustration (not runnable in the notebook; other tools have the same parts):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nowcast
spec:
  replicas: 2
  selector:
    matchLabels: {app: nowcast}
  template:
    metadata:
      labels: {app: nowcast}
    spec:
      containers:
        - name: server
          # an image digest (shortened here), not a movable tag
          image: registry.example.org/nowcast@sha256:4f6c0e2d
          ports: [{containerPort: 8000}]
          resources:
            requests: {cpu: "4", memory: 16Gi}
            limits: {memory: 16Gi, nvidia.com/gpu: 1}
          startupProbe:              # up to 60 x 10 s = 10 min to load
            httpGet: {path: /readyz, port: 8000}
            periodSeconds: 10
            failureThreshold: 60
          readinessProbe:
            httpGet: {path: /readyz, port: 8000}
            periodSeconds: 5
          livenessProbe:
            httpGet: {path: /healthz, port: 8000}
            periodSeconds: 10

3.4 Theorem: the autoscaling rule #

The Horizontal Pod Autoscaler (HPA) changes the number of replicas of a Deployment. Let $N$ be the current number of Pods, $m$ the current average metric per Pod (CPU utilisation as a fraction of the request, or requests per second per Pod) and $t$ the target value of the metric.

Theorem (HPA rule). The HPA asks for

replicas. If the load splits evenly over the Pods and does not change while scaling, this is the smallest number of Pods with an average metric of at most $t$.

Why this is true, in words. $N m$ is the total load. Spread it over $N'$ Pods and each gets $N m / N'$. Ask for this to be at most $t$, and take the smallest whole number.

Proof.

1. With $N$ Pods at average $m$, the total load is $L = N m$.
2. With $N'$ Pods and an even split, the average per Pod becomes $L / N'$.
3. We want $L / N' \le t$. Multiply by $N'/t > 0$: $N' \ge L/t = N m / t$.
4. The smallest integer that satisfies step 3 is $\lceil N m / t \rceil$. $\blacksquare$

The practical details (Kubernetes documentation, checked as of 2026; the values are the defaults and can be changed):

• Tolerance. The HPA does nothing while $|m/t - 1| \le 0.1$. This avoids changes for small noise.
• Scale-down stabilisation window. For scale-down, the HPA uses the highest recommendation of the last 300 s. So a short dip in traffic does not remove Pods that are needed a minute later. Scale-up has no window by default; it is limited to adding at most 4 Pods or 100% of the current Pods per 15 s, whichever is more.
• Pods without a metric count as 100% of the target in a scale-down and as 0% in a scale-up; Pods that are not yet ready count as 0% in a scale-up. Both rules make each step smaller.
• Several metrics: the HPA computes $N'$ for each and takes the largest.
• The HPA computes all this periodically, every 15 s by default.

Interactive figure: loading…

3.5 Kubernetes and an HPC batch scheduler #

Research clusters often use an HPC batch scheduler. The standard one is Slurm: you submit a script with sbatch that asks for nodes, GPUs and time; the job waits in a queue, runs once and ends. Kubernetes has the same building blocks, but it is built around long-running services.

Slurm	Kubernetes
a batch job (sbatch); a job step (srun)	a Job; its Pods
a job array (many similar jobs)	an Indexed Job
scrontab (scheduled jobs)	a CronJob
--gres=gpu:4	limits: {nvidia.com/gpu: 4} through the device plugin
--cpus-per-task, --mem	requests (scheduling) and limits (enforcement)
a partition (a group of nodes)	a node pool: node labels and taints, chosen with nodeSelector and tolerations
environment modules	the container image
(no equivalent)	Deployment + Service + probes + HPA for long-running services

Slurm is better at large multi-node training: gang scheduling (all nodes of a job start together or none does), placement that knows the network topology, and fair-share queues. Kubernetes needs add-ons for this (for example Kueue, JobSet or Volcano). Kubernetes is better at serving: self-healing replicas, rolling updates, load balancing and autoscaling.

3.6 Object storage #

Object storage (Amazon S3, Google Cloud Storage, Azure Blob Storage, and S3-compatible stores) is a map from keys to objects (blobs of bytes), used over HTTP. It is not a filesystem: s3://bucket/runs/42/ckpt.pt is one key, and the slashes are just characters. You can put, get, list and delete whole objects. The properties below are those of Amazon S3 general-purpose buckets, checked as of 2026; other stores can differ, so check yours.

1. Immutable objects. A write replaces a whole object. There is no append and no partial update. Large objects are uploaded in parts (multipart upload) and appear only when complete.
2. No atomic rename. A "move" is a copy plus a delete. It takes time proportional to the size and is not atomic: a reader can see both copies, or neither. (Since 2025, AWS offers an atomic rename only for the S3 Express One Zone storage class in directory buckets.)
3. Consistency. S3 has strong read-after-write consistency: after a successful PUT or DELETE, every later GET or LIST sees it. An update of one key is atomic: a reader sees the old object or the new one, never a mix. There is no atomic update across several keys, and two concurrent writers to one key end as "last writer wins". Conditional writes (If-None-Match, If-Match) allow "create only if absent".
4. Costs. You pay for storage (per GB-month), for requests (per 1,000 requests), and for egress: data that leaves the region or the cloud (per GB).

Layout for ML.

• Datasets: pack samples into immutable shards of about 100 MB to 1 GB (tar files as in WebDataset, Parquet files, Zarr chunks) under a versioned prefix, with a manifest that lists each shard and its hash. Millions of small objects are slow and expensive (worked example (c)).
• Checkpoints: one object per process ("rank") under a unique prefix, for example runs/<run_id>/step-0012000/rank-0003.pt. Write manifest.json last. The resume logic trusts only checkpoints that have a manifest. Never commit by "write to a temporary name, then rename": on object storage that rename is a slow, non-atomic copy.

3.7 Cost and capacity #

The cost of a job is

Cloud capacity comes in three types:

• on-demand: pay per hour, no commitment, no interruption by the provider;
• spot: spare capacity at a large discount (providers advertise up to about 90%), but the provider can take it back. As of 2026, AWS gives a 2-minute notice; Google Cloud by default gives only a best-effort shutdown period of up to 30 s;
• reserved or committed: a 1- to 3-year commitment at a discount; you pay for every hour, used or not.

Result (reserved break-even). Let $p_{\text{od}}$ be the on-demand price per hour, $p_{\text{res}}$ the effective reserved price per hour, and $u \in (0, 1]$ the share of hours you really use. Reserving is cheaper exactly when $u > p_{\text{res}} / p_{\text{od}}$.

Proof. Over $H$ hours, reserved capacity costs $p_{\text{res}} H$ and gives $uH$ used hours, so it costs $p_{\text{res}}/u$ per used hour. On-demand costs $p_{\text{od}}$ per used hour. $p_{\text{res}}/u < p_{\text{od}} \iff u > p_{\text{res}}/p_{\text{od}}$. $\blacksquare$

Result (multi-node jobs are interrupted more often). Let each of $k$ nodes be interrupted independently, at random times (a Poisson process) with mean time $M_{\text{node}}$ between interruptions. A job that needs all $k$ nodes stops when any node is interrupted. Its mean time between interruptions is

Proof.

1. For one node, the time to the next interruption $T_i$ is exponential with rate $1/M_{\text{node}}$: $P(T_i > s) = e^{-s/M_{\text{node}}}$ (the memoryless property of a Poisson process).
2. The job runs past time $s$ only if every node does. By independence, $P(\min_i T_i > s) = \prod_{i=1}^{k} e^{-s/M_{\text{node}}} = e^{-ks/M_{\text{node}}}$.
3. This is an exponential distribution with rate $k/M_{\text{node}}$, so its mean is $M_{\text{node}}/k$. $\blacksquare$

3.8 Theorem: Young's checkpoint interval #

A long training job on spot nodes saves a checkpoint (the model and optimiser state) every $\tau$ minutes of work. Saving takes time $C$. When an interruption comes, all work since the last checkpoint is lost. Checkpoint too often and you waste time saving; checkpoint too rarely and you lose a lot of work at each interruption.

Theorem (Young, 1974). Let one checkpoint take time $C$, and let interruptions form a Poisson process with mean time $M$ between them. Checkpointing every $\tau$ of work wastes, to first order, a fraction

of the useful time. It is smallest at

A restart time $R$ (start new nodes, load the checkpoint) adds $R/M$, which does not depend on $\tau$.

Why this is true, in words. The first term falls as $\tau$ grows; the second rises. The best $\tau$ is where they balance, and there the two terms are equal.

Proof (first order, for $C \ll \tau \ll M$).

1. Every $\tau$ of work costs $C$ extra: an overhead fraction $C/\tau$.
2. An interruption falls at a uniformly random point of the current interval (a Poisson process has no memory), so on average $\tau/2$ of work is lost. Interruptions come at rate $1/M$ per unit of time, so the lost fraction is $(1/M) \cdot (\tau/2) = \tau/(2M)$.
3. Add the two: $w(\tau) = C/\tau + \tau/(2M)$.
4. Set the derivative to zero: $w'(\tau) = -C/\tau^2 + 1/(2M) = 0$, so $\tau^2 = 2CM$ and $\tau^* = \sqrt{2CM}$. Since $w''(\tau) = 2C/\tau^3 > 0$, this is a minimum.
5. At $\tau^*$: $C/\tau^* = C/\sqrt{2CM} = \sqrt{C/(2M)}$ and $\tau^*/(2M) = \sqrt{2CM}/(2M) = \sqrt{C/(2M)}$. The two terms are equal, and $w(\tau^*) = 2\sqrt{C/(2M)} = \sqrt{2C/M}$. $\blacksquare$

Interactive figure: loading…

When the first-order assumptions fail

The proof assumes $C \ll \tau \ll M$. It ignores interruptions that hit a checkpoint write or a restart, and work lost more than once in one interval. When $\tau^*$ is not much smaller than $M$ (for example a large job on unreliable nodes), the real waste is larger than $w(\tau^*)$, and the best interval moves a little. The simulation in the widget and in the notebook counts all these effects. Daly (2006) derives higher-order estimates of the best interval for this case. If $w(\tau^*)$ is large (say above 20%), fix the inputs instead: write checkpoints faster (asynchronous or sharded checkpoints make $C$ smaller), or use fewer or more reliable nodes (larger $M$).

Answers to section 1. (1) It reads the rows of the failed attempt plus the rows of the retry: duplicates. Appending is not idempotent (section 3.1). (2) $\lceil 3 \cdot 85/50 \rceil = \lceil 5.1 \rceil = 6$ replicas (section 3.4). (3) Once every $24/8 = 3$ hours on average (section 3.7): eight times as often as one node.

4. Worked example #

A ground-motion surrogate (a neural network that predicts earthquake shaking much faster than a physics simulation) is fine-tuned on cloud GPUs and then served on Kubernetes. All prices are assumptions of the order of public list prices, not quotes.

(a) Autoscaling the server. The HPA targets $t = 80$ requests/s per Pod.

• 5 replicas at an average of 120 requests/s: $m/t = 120/80 = 1.5$, outside the tolerance, so $N' = \lceil 5 \cdot 1.5 \rceil = \lceil 7.5 \rceil = 8$.
• 8 replicas at 76 requests/s: $m/t = 0.95$ and $|0.95 - 1| = 0.05 \le 0.1$, so no change.
• 8 replicas at 50 requests/s: $m/t = 0.625$, so $N' = \lceil 8 \cdot 0.625 \rceil = 5$. This is a scale-down, so it happens only when 5 is the highest recommendation of the last 300 s.

(b) Training on spot. The job uses 4 nodes × 8 GPUs for 8 hours: 256 GPU-hours. Prices: $3.00 per GPU-hour on-demand, $1.10 on spot. A checkpoint is 60 GB written at 500 MB/s: $C = 60{,}000/500 = 120$ s $= 2$ min. Each node is interrupted on average once per 24 h. A restart costs $R = 5$ min.

1. Mean time between interruptions of the job: $M = 24/4 = 6$ h $= 360$ min.
2. Interval: $\tau^* = \sqrt{2 \cdot 2 \cdot 360} = \sqrt{1440} = 37.9$ min.
3. Waste: $\sqrt{2 \cdot 2/360} = 0.105$, plus $R/M = 5/360 = 0.014$: about $w = 0.119$, or 11.9%. (The simulation in section 5 gives about 12.8%, with the best interval near 38 min.)
4. Compute: on-demand $256 \times 3.00 = $768.00$; spot $256 \times 1.119 \times 1.10 = $315.19$.
5. Storage and egress (the same for both): the last 3 checkpoints, 180 GB for a month at $0.023 per GB-month, cost $4.14. Downloading the final 60 GB at $0.09 per GB costs $5.40.
6. Totals: on-demand $777.54, spot $324.73, a saving of 58%. Spot stays cheaper while $(1 + w) \cdot 1.10 < 3.00$, that is, up to a waste of $w = 1.73$ (173%).

The AWS notice (2 min) equals the checkpoint write time, with no margin, and the default Google Cloud shutdown period (up to 30 s) is shorter. So checkpoint periodically and resume automatically; use the notice only to stop cleanly.

(c) Small files vs shards. A dataset has 10 million samples of 10 KB each (100 GB).

• As 10 million separate objects: at $0.0004 per 1,000 GET requests, one epoch costs $10^7/1000 \times 0.0004 = $4.00$ in requests alone. At 20 ms per request, reading them one after the other takes $10^7 \times 0.02$ s $= 200{,}000$ s $\approx 55.6$ hours; with 64 parallel requests, about 52 minutes.
• As 1,000 shards of 100 MB: 1,000 requests cost $0.0004 per epoch, and the time is set by the bandwidth, not by the number of requests.

5. In code #

Plain Python checks the numbers of section 4 (no cluster needed). The notebook has these cells and the exercises.

import math
import numpy as np

# (a) the HPA rule with its default tolerance
def hpa_desired(current, metric, target, tolerance=0.1):
    ratio = metric / target
    return current if abs(ratio - 1) <= tolerance else math.ceil(current * ratio)

print(hpa_desired(5, 120, 80), hpa_desired(8, 76, 80), hpa_desired(8, 50, 80))   # 8 8 5

# (b) Young's interval (minutes): checkpoint C, restart R, mean gap M, W minutes of work
C, R, M, W = 2.0, 5.0, 360.0, 9600.0
print(round(math.sqrt(2 * C * M), 1), round(math.sqrt(2 * C / M) + R / M, 3))   # 37.9 0.119

def overhead(tau, runs=200, seed=0):
    """Monte Carlo waste fraction with Poisson interruptions."""
    rng = np.random.default_rng(seed)
    total = 0.0
    for _ in range(runs):
        done, wall, next_stop = 0.0, 0.0, rng.exponential(M)
        while done < W:
            piece = min(tau, W - done)
            need = piece + (C if done + piece < W else 0.0)   # no checkpoint after the last piece
            if wall + need <= next_stop:                      # the piece and its checkpoint finish
                wall, done = wall + need, done + piece
            else:                                             # interrupted: lose the piece, restart
                wall = next_stop + R
                next_stop = wall + rng.exponential(M)
        total += wall / W - 1
    return total / runs

for tau in (10, 20, 38, 70, 100):
    print(tau, round(overhead(tau), 3))
# 10 0.237
# 20 0.15
# 38 0.128
# 70 0.153
# 100 0.194

# (b) on-demand vs spot for 256 GPU-hours (prices are assumptions, USD)
w = math.sqrt(2 * C / M) + R / M
storage, egress = 3 * 60 * 0.023, 60 * 0.09
print(round(256 * 3.00 + storage + egress, 2), round(256 * (1 + w) * 1.10 + storage + egress, 2))
# 777.54 324.73

The simulated waste is a little above the first-order formula (12.8% vs 11.9%), because it also counts interruptions during checkpoint writes and restarts. The minimum is flat: intervals from about 30 to 50 minutes cost within about one percentage point of the best.

6. Where it shows up #

• Operational weather and climate ML. A forecast service is a DAG triggered by new analysis data: ingest, check the data contract (lesson 01), run inference, write products. A rerun after a failure must not duplicate products, so every task is keyed by the analysis time and commits with a manifest.
• Industry pipelines. A fraud or recommendation model is retrained nightly by a DAG and backfilled after every bug fix in the feature code. Backfills are safe only with idempotent tasks.
• Serving on Kubernetes. Model servers run as Deployments behind a Service, with probes and an HPA on CPU, GPU or requests per second. Serving frameworks such as KServe and Ray Serve add scale-to-zero and model routing on top of the same objects.
• HPC and cloud together. The same image trains on an HPC cluster with Slurm and serves on Kubernetes. Large reanalysis datasets (for example ERA5) stay in object storage near the compute, because the egress of many terabytes can cost more than the GPUs.

7. Common mistakes #

• now() inside a scheduled task, or appending in a task that can be retried. Use the data interval, replace the output, and commit with a manifest.
• A liveness probe that checks the model or a database. Slow loads become restart loops, and one database outage restarts every replica. Liveness checks only the process; readiness checks the model; a startup probe covers the slow start.
• No CPU request. The HPA has no utilisation to work with. A memory limit below the real peak gives OOM kills.
• Deploying a movable tag such as latest instead of a digest: Pods on different nodes can run different images, and rollback is not reliable.
• Rename-based commits on object storage, millions of small objects, or listing huge prefixes on every read.
• Saving a large checkpoint only when the interruption notice arrives. The notice can be shorter than the write.
• Planning a multi-node job with the per-node interruption rate. With $k$ nodes, interruptions come $k$ times as often.

8. Check yourself #

1. 4 replicas run at 30 requests/s each; the target is 50 requests/s per Pod. The load stays like this for 10 minutes. What does the HPA do, and when?
   Answer

   $m/t = 30/50 = 0.6$, outside the tolerance, so $N' = \lceil 4 \cdot 0.6 \rceil = \lceil 2.4 \rceil = 3$. This is a scale-down, so it waits until 3 is the highest recommendation of the last 300 s (default window): about 5 minutes after the load dropped. Check: 3 Pods at $120/3 = 40$ requests/s each, below the target.
2. Give the three probes, what each checks, and what Kubernetes does when each fails.
   Answer

   Liveness: is the process alive? Failure → the container is restarted. Readiness: may the Pod get traffic? Failure → the Pod is removed from the Service's endpoints, no restart. Startup: has the container finished starting? While it has not succeeded, the other two do not run; if it fails for longer than its budget (`failureThreshold` × `periodSeconds`), the container is restarted.
3. A server loads its model in 4 minutes. Its startup probe has periodSeconds: 10 and failureThreshold: 18. Is this enough?
   Answer

   No. The budget is $18 \times 10 = 180$ s, less than the 240 s load, so the container is restarted before it can finish: a restart loop. Use for example `failureThreshold: 30` (300 s), with some margin above the worst load time.
4. A forecast task writes its parts directly to s3://fc/2026-01-03T00/part-<i>.nc, and the next task reads every object under that prefix. Make it idempotent.
   Answer

   Each attempt writes its parts under its own prefix, for example `s3://fc/2026-01-03T00/attempt=/part-.nc`. When all parts are written, it writes `s3://fc/2026-01-03T00/manifest.json` last, listing exactly those parts. One PUT of one key is atomic, so a reader sees the old manifest or the new one. The reader reads only the parts named in the manifest. Old attempts are deleted later by a cleanup job.
5. A job runs on 16 spot nodes; each node is interrupted on average once every 32 hours. A checkpoint takes 1 minute. Give $M$, $\tau^*$ and the waste (without restart time).
   Answer

   $M = 32/16 = 2$ h $= 120$ min. $\tau^* = \sqrt{2 \cdot 1 \cdot 120} = \sqrt{240} \approx 15.5$ min. Waste $\sqrt{2 \cdot 1/120} \approx 0.129$, about 12.9%. Here $\tau^*/M \approx 0.13$, so the first-order formula is still reasonable.
6. Why does the HPA measure CPU utilisation relative to the request and not to the node?
   Answer

   The request is what the scheduler reserved for the Pod, so "100% of the request" means "the Pod uses what it was given". Pods of many sizes share a node, so utilisation of the whole node says nothing about one Deployment. It also means that a Pod without a CPU request has no utilisation, and the HPA cannot scale it on CPU.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. When is a task idempotent?

   Show answer

2. How do you commit a multi-part output safely on object storage?

   Show answer

3. What is the difference between requests and limits in Kubernetes?

   Show answer

4. What does Kubernetes do when a liveness probe fails, and when a readiness probe fails?

   Show answer

5. What is a startup probe for?

   Show answer

6. Give the HPA rule and compute it for 3 Pods at 85% with target 50%.

   Show answer

7. Why is "write to a temporary name, then rename" unsafe on object storage?

   Show answer

8. A job needs all of its $k$ nodes. Each node is interrupted with mean gap $M_{\text{node}}$. What is the job's mean gap?

   Show answer

9. State Young's checkpoint interval and the waste at the optimum.

   Show answer

10. When does reserved capacity beat on-demand?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "What happens when you deploy a new model version on Kubernetes?" — The Deployment gets a new image digest. A rolling update starts new Pods. The startup and readiness probes gate the traffic; the Service sends requests only to ready Pods; old Pods drain and stop. If something goes wrong, roll back to the previous ReplicaSet (kubectl rollout undo).
• "Liveness vs readiness?" — Restart the container vs remove the Pod from the load balancer. Never point liveness at the model or at a dependency.
• "Your nightly pipeline failed halfway. How do you rerun it safely?" — Tasks keyed by the data interval, outputs replaced by partition, commits with a manifest written last, no now(). Then a rerun or a backfill gives the same result as one clean run.
• "Your training runs on spot GPUs. Make it robust and estimate the cost." — Periodic sharded checkpoints at about $\sqrt{2CM}$ with $M = M_{\text{node}}/k$, a manifest per checkpoint, automatic resume; cost = GPU-hours × (1 + waste) × price + storage + egress.

10. Practice #

The notebook for this lesson runs in Colab. It has four exercises, each with a check cell and a folded solution:

1. hpa_desired with a tolerance, then an HPA simulation over a load curve with a scale-down window.
2. An idempotent daily task that writes to a temporary directory with an attempt prefix and a manifest.json written last, and a reader that trusts only the manifest; test crash and retry.
3. overhead(tau, C, R, M, W, runs, seed): a Monte Carlo of checkpointing under Poisson interruptions; compare it with $C/\tau + \tau/(2M) + R/M$ and find the best $\tau$.
4. job_cost(...): on-demand vs spot with storage and egress, and the waste at which spot stops being cheaper.

11. Watch and read #

Kubernetes Explained in 6 Minutes | k8s ArchitectureByteByteGo · 6 min

The control plane, nodes, Pods and the control loop that keeps the desired state.

Kubernetes Health Checks: Liveness vs. Readiness vs. Startup ProbeAnton Putra · 3 min

What each probe does when it fails; compare with the table in section 3.3.

The Importance of Idempotency in Designing Data PipelinesPracticalGCP · 3 min

Why reruns of pipeline steps must give the same result.

• Kubernetes documentation: Pods, Deployments, Liveness, readiness and startup probes, Resource management for Pods and containers, and Horizontal Pod Autoscaling (sections "Algorithm details" and "Stabilization window").
• Apache Airflow documentation, DAG Runs: data intervals, catchup and backfill.
• Amazon S3 User Guide, What is Amazon S3?, section "Amazon S3 data consistency model".
• AWS, Spot Instance interruption notices; Google Cloud, Spot VMs.
• J. W. Young, "A first order approximation to the optimum checkpoint interval", Communications of the ACM 17(9), 1974, doi:10.1145/361147.361115; J. T. Daly, "A higher order estimate of the optimum checkpoint interval for restart dumps", Future Generation Computer Systems 22(3), 2006, doi:10.1016/j.future.2004.11.016.