Containers, GPU images and CI pipelines

Before this lesson: lesson 01 (the environment is an input of a run; digest vs tag) and lesson 02 (the tests that CI runs). You need a terminal and the idea of a dependency file that pins package versions (a lockfile). For the GPU part, ML systems 01 helps but is not required. No math in this lesson. Time: about 55 minutes, plus the notebook.

Why this matters #

In lesson 01 the environment was one of the inputs of a training run, and a container image digest was the best way to version it. This lesson shows how to make that image, how to make it run on GPUs, and how to build and test it automatically.

• Reproducibility. The image digest pins every library. A result from last year can be rebuilt with the same image.
• One artifact everywhere. The same image runs on a laptop, in CI, on a Kubernetes cluster (lesson 04), behind a serving API (lesson 05) and on an HPC cluster.
• Fast, safe changes. CI runs the tests of lesson 02 on every pull request in a few minutes. The release lesson (lesson 06) starts from the image that CI built.
• Interviews. "How do you package a model?", "Why is my Docker build slow?" and "How does a container get the GPU?" are common ML engineering questions.

1. Predict first #

2. The picture #

A container is a process #

A container is a normal process on the host's Linux kernel, with two kinds of isolation:

• namespaces give it its own view of the system: its own file system, process ids, network and users;
• cgroups (control groups) limit the resources it can use: CPU, memory, devices.

A virtual machine (VM) is different: it runs a whole guest operating system with its own kernel on emulated hardware. A container starts in a fraction of a second and adds almost no overhead, because it shares the host kernel. The price: it cannot bring its own kernel, and, as we will see, it cannot bring its own GPU kernel driver.

The file system of a container comes from an image: a stack of read-only layers (OS libraries, Python packages, your code) plus a small configuration (the command to run, environment variables). The image is built from a recipe, the Dockerfile, one instruction per line.

The build cache #

The builder runs the instructions from top to bottom. For each one it checks its cache: if the same instruction, with the same inputs, ran before on top of the same earlier steps, the old result is reused. Once one step misses the cache, every later step runs again.

The GPU stack #

A GPU program in a container uses four parts. Two come from the host, two from the image:

A CI pipeline is a graph of jobs #

A CI pipeline is a set of jobs (lint, unit tests, data tests, a smoke training, the image build) with "needs" edges between them. Jobs without an edge run in parallel. A job runs only when every job it needs has passed.

3. Definitions and main results #

3.1 Images, layers, digests #

• A layer is a set of file changes (added, changed or deleted files). RUN, COPY and ADD add layers; other instructions (WORKDIR, ENV, CMD) only change the image's configuration. All of them are build steps that go through the cache.
• The digest of an image is the SHA-256 hash of its manifest, a small JSON document that lists the configuration and the digest of every layer. If any layer changes, the digest changes. It is written image@sha256:4f6c….
• A tag (nowcast:latest, nowcast:v3) is a movable name. Anyone who pushes can make it point to a new digest tomorrow. Deploy by digest, or by a tag that is never reused, such as the commit SHA.
• A registry (Docker Hub, GitHub Container Registry, a cloud registry) stores images by digest and keeps the tags.
• The build context is the folder sent to the builder. A .dockerignore file keeps data, checkpoints, virtual environments and .git out of it. Without it, COPY . . copies them, makes the image large, and misses the cache whenever any of them changes.

3.2 The cache rule and the order of the steps #

Cache rule. A step is reused only if (1) its instruction is unchanged, (2) its inputs are unchanged (for COPY, the checksums of the copied files) and (3) the step before it was reused.

Result (order by change frequency). Put the steps whose inputs change rarely first and the steps whose inputs change often last. In particular: copy the dependency file, install, and only then copy the code. Then a code change reruns only the code copy.

Proof.

1. A code change modifies the inputs of the first COPY that includes the code.
2. By condition (3), that step and every later step miss the cache.
3. If pip install comes after COPY . /app, it is a later step, so every code change reinstalls all packages.
4. If only requirements.lock is copied before the install, the inputs of both steps are unchanged by a code change, and so are all earlier steps. They are reused. Only COPY src/ and the steps after it run again. $\blacksquare$

The same argument says: a step with an expensive output and rare changes (system packages, Python packages) belongs high in the file; your code belongs at the bottom.

Deleting files in a later step does not shrink the image. Layers are read-only. If one RUN downloads 2 GB and a later RUN deletes it, the first layer still holds the 2 GB; the second layer only marks the files as deleted. Clean up in the same RUN that created the files.

Multi-stage builds. A build needs compilers, headers and download caches; the running model does not. A multi-stage build installs everything in a first stage and copies only the result into a small second stage. The final image contains only the last stage:

# syntax=docker/dockerfile:1
# ---- stage 1 (build): compilers and caches stay here
FROM python:3.12 AS build
RUN python -m venv /opt/venv
ENV PATH=/opt/venv/bin:$PATH
COPY requirements.lock /tmp/requirements.lock
RUN pip install --no-cache-dir -r /tmp/requirements.lock

# ---- stage 2 (runtime): the interpreter, the environment and the code
FROM python:3.12-slim
COPY --from=build /opt/venv /opt/venv
ENV PATH=/opt/venv/bin:$PATH
WORKDIR /app
COPY src/ ./src/
USER 1000
CMD ["python", "-m", "src.serve"]

A code change reruns only COPY src/; a new pin in requirements.lock reruns the install. USER 1000 runs the server as a normal user, not as root. Other build tools (Podman, Buildah, Kaniko, BuildKit) read the same Dockerfile format and follow the same cache rule.

3.3 GPU containers and the compatibility rule #

The CUDA stack has four parts:

Each driver branch supports CUDA up to one version: 535 supports CUDA 12.2, 570 supports 12.8, 580 supports 13.0. The compatibility rule says whether an image built with CUDA version $v$ runs on a driver:

• Backward compatibility. A newer driver runs software built with any older CUDA version. A CUDA 11.8 image runs on driver 580.
• Minor-version compatibility (since CUDA 11). Software built with any CUDA 12.x runs on a driver $\ge 525$; with any 13.x, on a driver $\ge 580$. So a 12.8 image runs on driver 535 (a 12.2 driver). Two limits: no PTX JIT (the driver cannot compile the portable PTX code of a newer CUDA, so the image must contain machine code for the GPU architecture), and features that need a newer driver return an error (cudaErrorCallRequiresNewerDriver).
• Forward compatibility. For a newer major version (a 13.0 image on driver 535 or 570), or for PTX JIT inside a major version, install the package cuda-compat-<major>-<minor>. It provides a newer user-mode driver libcuda.so (found through LD_LIBRARY_PATH) that works on top of the older kernel-mode driver. It is supported only on data-center GPUs (and a few other platforms), and only on the driver branches in NVIDIA's table.

Why this is true, in one sentence. The kernel-mode driver cannot be in the image, so the image must ask for nothing that the host's driver cannot do; minor-version compatibility and the compat package are two ways NVIDIA lets a newer image talk to an older driver.

3.4 Containers on shared clusters #

The Docker daemon normally runs as root. A user who can start Docker containers can get root on the host, so HPC centres do not allow it. Rootless container runtimes run the same image with the user's own permissions:

• Apptainer (formerly Singularity) converts the image to one file (SIF); the flag --nv mounts the host's NVIDIA driver libraries into the container.
• Enroot imports the image as a file system; Pyxis, a plugin for the Slurm scheduler, adds srun --container-image=....
• Podman runs containers without a daemon and without root.

Slurm is the standard batch scheduler on HPC clusters: you submit a job script, it waits in a queue, and it runs when nodes are free. All these runtimes run images in the standard OCI format (Open Container Initiative), the format that Docker builds. So one image runs on a laptop, in CI, on Kubernetes and on an HPC cluster; only the runtime and the mounts (data folders, the GPU driver, the fast network libraries of the cluster) change.

3.5 CI for an ML repository #

Continuous integration (CI) runs automatic checks on every change before it is merged. The terms, with GitHub Actions as the example (GitLab CI, Jenkins and others have the same parts):

• Workflow: a YAML file in the repository (.github/workflows/ci.yml) with one or more jobs.
• Trigger (event): what starts a run: a pull request, a push to main, a schedule (cron), or a manual start.
• Job: a list of steps that runs on one runner. Jobs run in parallel unless needs: orders them. If a needed job fails, the jobs that need it are skipped. Jobs share no files: pass files with artifacts (upload in one job, download in the next) or a cache.
• Step: one shell command (run:) or one reusable action (uses:).
• Runner: the machine that runs a job. A hosted runner is a fresh virtual machine for each job; standard hosted runners have no GPU. A self-hosted runner is your own machine, for example a GPU server.
• Cache: saved files reused between runs, with a key such as the hash of the lockfile: the downloaded packages, the image layers.
• Secret: an encrypted value (a registry password, a cloud key) given to the job at run time and hidden in logs. Secrets are not given to workflows started from a fork of the repository. Better than a long-lived key: OIDC (OpenID Connect), where the cloud trusts a short-lived token that the CI system issues for one run.

A short workflow. The action versions are the latest major versions as of November 2026; check them when you copy the file.

# .github/workflows/ci.yml
name: ci
on:
  pull_request:                         # every pull request: fast checks, < 10 minutes
  push:
    branches: [main]                    # after a merge: also build and push the image
jobs:
  test:
    runs-on: ubuntu-latest              # hosted runner: a fresh VM for this job
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
          cache: pip                    # dependency cache ...
          cache-dependency-path: requirements.lock   # ... keyed on the lockfile's hash
      - run: pip install -r requirements.lock
      - run: ruff check .                                     # lint
      - run: pytest tests/unit tests/data tests/model -q      # unit, data-contract, behavioural
      - run: python -m nowcast.train --smoke --max-steps 20   # loss falls, no NaN
  image:
    needs: test                         # starts only if "test" passed
    if: github.event_name == 'push'     # only after a merge to main
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write                   # lets the built-in token push to the registry
    steps:
      - uses: actions/checkout@v7
      - uses: docker/login-action@v4
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - uses: docker/setup-buildx-action@v4
      - uses: docker/build-push-action@v7
        with:
          push: true
          tags: ghcr.io/example-lab/nowcast:${{ github.sha }}   # never reused: the commit SHA
          cache-from: type=gha          # image layer cache
          cache-to: type=gha,mode=max

What runs when:

Build once, deploy the same digest. The image that passed the tests is the image that goes to staging and production. Rebuilding it later for production can pull other package versions.

Answers to section 1. (1) Yes. COPY . /app has new inputs, so it and every later step run again, including the install. Copy the dependency file first (section 3.2). (2) The CUDA 12.8 image runs on driver 535 through minor-version compatibility (12.x needs $\ge 525$), unless it needs PTX JIT or a feature of a newer driver. The CUDA 13.0 image does not run (13.x needs $\ge 580$), except with the forward-compatibility package on a data-center GPU. (3) No. Jobs run on separate runners and share no files; use an artifact or a cache.

4. Worked example #

(a) One Dockerfile in two orders. Assumed times: downloading the base image takes 30 s, installing the packages 240 s, each COPY 1–2 s, WORKDIR and CMD almost 0 s.

Apply the cache rule to four changes. For each one, find the first step whose inputs changed; it and every later step run again.

Step by step for the first row, code first: step 1 (FROM) has the same base, so it is reused. Step 2 has no file inputs and its earlier step was reused, so it is reused. Step 3 copies src/train.py, which changed: first miss. Steps 4 and 5 rerun by condition (3). With the lockfile first, the first file input that changed is in step 5. On a team that changes code 50 times a day, that is about 3.4 hours of builds a day against 2 minutes.

(b) The CI pipeline and its caches. Four test jobs (lint 0.5 min, unit 2 min, data 1 min, smoke training 3 min) run in parallel; each also needs a setup (checkout and install) of 3 min without a cache and 0.5 min with a cache hit. The image job needs all four tests and takes 5 min without the layer cache and 1 min with it.

• Wall time without caches: the slowest test job takes $3 + 3 = 6$ min, then the image job 5 min: 11 min. Runner time (what you pay for): $3.5 + 5 + 4 + 6 + 5 = 23.5$ min.
• With cache hits: $0.5 + 3 = 3.5$ min, then 1 min: 4.5 min. Runner time: $1 + 2.5 + 1.5 + 3.5 + 1 = 9.5$ min.
• If the unit tests fail, the image job is skipped (it needs unit); the other test jobs still run to the end, so you see all failures of the change at once.

5. In code #

Docker and a CI service are not available in a notebook, but their rules are small programs. Here the cache rule, an image digest, and the plan of a CI pipeline in plain Python (graphlib is in the standard library):

import hashlib, json
from graphlib import TopologicalSorter

def rebuilt_steps(steps, changed):
    """steps: list of (instruction, input files). Indices of the steps that run again."""
    out, miss = [], False
    for i, (_, inputs) in enumerate(steps):
        miss = miss or bool(set(inputs) & set(changed))   # first miss, then every later step
        if miss:
            out.append(i)
    return out

code_first = [("FROM python:3.12-slim", {"base"}), ("WORKDIR /app", set()),
              ("COPY . .", {"src/", "requirements.lock", "README.md"}),
              ("RUN pip install -r requirements.lock", set()), ("CMD [...]", set())]
lock_first = [("FROM python:3.12-slim", {"base"}), ("WORKDIR /app", set()),
              ("COPY requirements.lock .", {"requirements.lock"}),
              ("RUN pip install -r requirements.lock", set()),
              ("COPY src/ ./src/", {"src/"}), ("CMD [...]", set())]
print(rebuilt_steps(code_first, ["src/"]), rebuilt_steps(lock_first, ["src/"]))   # [2, 3, 4] [4, 5]

# The digest is a hash of the manifest, which lists the layer digests
def image_digest(layer_digests):
    manifest = json.dumps({"layers": layer_digests}, sort_keys=True)
    return "sha256:" + hashlib.sha256(manifest.encode()).hexdigest()
v1 = [hashlib.sha256(x).hexdigest() for x in [b"base", b"deps-v1", b"code-v1"]]
v2 = v1[:2] + [hashlib.sha256(b"code-v2").hexdigest()]
print(image_digest(v1) != image_digest(v2))   # True: one new layer, a new digest

# A CI plan: jobs in dependency order; a job runs only if all the jobs it needs passed
needs = {"lint": [], "unit": [], "data": [], "smoke": [],
         "image": ["lint", "unit", "data", "smoke"], "push": ["image"]}
state = {}
for job in TopologicalSorter(needs).static_order():
    if any(state[d] != "passed" for d in needs[job]):
        state[job] = "skipped"
    else:
        state[job] = "failed" if job == "unit" else "passed"
print(state)
# {'lint': 'passed', 'unit': 'failed', 'data': 'passed', 'smoke': 'passed',
#  'image': 'skipped', 'push': 'skipped'}

6. Where it shows up #

• Every production model. Serving platforms (Kubernetes, cloud model endpoints, serverless containers) take an image as their unit of deployment. Model servers such as NVIDIA Triton and vLLM are distributed as images.
• Training at scale. NVIDIA's NGC images bundle a tested CUDA, cuDNN, NCCL and PyTorch; teams build their training image on top of one. On HPC clusters the same image runs through Apptainer or Enroot/Pyxis, with the cluster's network libraries mounted in.
• ML for science. Weather and climate centres package ML forecast models as images so that a research result and the operational run use the same environment. The driver rule matters there: supercomputers often update their drivers on a slower schedule than the CUDA versions that new frameworks need.
• CI in ML teams. Pull request checks with a smoke training and data-contract tests, nightly or weekly full training on a GPU cluster, and the image built once per merge are a common pattern (Google's "MLOps: continuous delivery and automation pipelines in ML" calls the automated retraining part "continuous training").

7. Common mistakes #

• COPY . . before the dependency install, so every code change reinstalls all packages.
• No .dockerignore: data, checkpoints and .git go into the image and break the cache.
• Deleting files in a later layer to "make the image smaller". Clean up in the same RUN.
• Deploying a movable tag (latest) instead of a digest or a never-reused tag: no reproducibility and no reliable rollback.
• Expecting the image to bring the GPU driver, or using a newer major CUDA than the host driver supports.
• Full training in pull request CI (slow, expensive, and hosted runners have no GPU), or no model test in CI at all.
• Secrets in the repository or printed in logs, or long-lived cloud keys where short-lived OIDC credentials would work.

8. Check yourself #

1. In the multi-stage Dockerfile of section 3.2, which steps run again when requirements.lock changes, and why is the final image smaller than the build stage?
   Answer

   In the build stage: the `COPY requirements.lock` step, the install and every later step. In the runtime stage: `COPY --from=build` (its input, the virtual environment, changed) and every later step. The runtime stage starts from the slim base image and copies only `/opt/venv` and the code; compilers, headers and download caches stay in the build stage, which is not part of the final image.
2. A teammate says: "Our image is tagged nowcast:prod, so we know what runs in production." What is wrong, and what do you log instead?
   Answer

   A tag can be moved to a new image at any time, so `prod` today may not be `prod` of last week. Log and deploy the digest (`nowcast@sha256:...`), or a tag that is never reused, such as the commit SHA.
3. A node has driver 535. Which of these images run: CUDA 11.8, 12.2, 12.8 (needs PTX JIT), 13.0 on a data-center GPU with cuda-compat-13-0?
   Answer

   11.8: yes (backward compatibility). 12.2: yes (same version as the driver). 12.8 with PTX JIT: no, minor-version compatibility does not allow PTX JIT; it runs if the image has machine code for the GPU, or with the forward-compatibility package. 13.0 with `cuda-compat-13-0` on a data-center GPU: yes, NVIDIA lists this package as compatible with the 535 branch.
4. Why may a cluster allow Apptainer but not Docker?
   Answer

   The Docker daemon runs as root, and a user who can start containers through it can get root on the host. Apptainer (and Enroot, Podman) run the container with the user's own permissions, and they run the same OCI images.
5. Your pull request pipeline takes 25 minutes. Name three ways to make it faster without testing less.
   Answer

   Cache the dependencies (key on the lockfile hash) and the image layers; run independent jobs in parallel instead of in one long job; order the Dockerfile by change frequency; use small fixture files and a few smoke-training steps instead of real data; move the full training to a scheduled pipeline.

9. Interview angle #

• "Why is my Docker build slow, and how do you fix it?" — The cache rule: find the first step that misses; copy dependency files and install before copying the code; add a .dockerignore; use a layer cache in CI; multi-stage build for a small image.
• "How does a container get the GPU?" — The orchestrator exposes the device (a Kubernetes device plugin or the runtime's GPU flag); the NVIDIA Container Toolkit mounts the device files and the host's user-mode driver; the image brings the CUDA runtime and libraries; the host driver must be new enough for the image's CUDA.
• "What does your CI pipeline for an ML repo look like?" — The table of section 3.5: fast checks on every pull request, build once and push by commit SHA after a merge, scheduled full training on a GPU cluster with evaluation and registration.
• "How do you handle credentials in CI?" — Secrets from the CI's secret store, never in the repository; not available to forks; short-lived OIDC credentials for cloud access; least permissions per job.

10. Practice #

The notebook for this lesson runs in Colab. It has four exercises, each with a check cell and a folded solution. Everything is plain Python; no Docker is needed.

1. Write rebuilt_steps(steps, changed) (the cache rule) and compare the two orders of a Dockerfile for several changes.
2. Write image_digest(layers), push and resolve for a tiny registry, and show that a tag can point to two different digests while a digest reference never moves.
3. Write cuda_runs(driver, cuda, ...) from the compatibility rule and a small verified table, and test the edge cases.
4. Write ci_plan(needs, minutes, active, failed) with graphlib: which jobs run, pass, fail or are skipped, and the wall time of the pipeline.

11. Watch and read #

• Docker documentation, Build cache and Multi-stage builds.
• NVIDIA, CUDA Compatibility: sections "Minor Version Compatibility" and "Forward Compatibility" (with the table of supported driver branches).
• Apptainer documentation, GPU support (the --nv flag).
• GitHub documentation, Understanding GitHub Actions and Events that trigger workflows (secrets and forks).
• Made With ML, CI/CD for machine learning.
• Google Cloud, MLOps: continuous delivery and automation pipelines in machine learning.