Serving APIs, dynamic batching and queueing latency

Before this lesson: lesson 04 (Pods, probes, replicas). From probability: the exponential and Poisson distributions and the geometric series (probability lesson 03); the proof in section 3.3 uses a Markov chain in continuous time, explained there in plain words (probability lesson 15 has more). Time: about 75 minutes, plus the notebook.

---

Why this matters #

A model is useful only when other programs can call it: a ranking model behind a search page, a fraud model inside a payment, a nowcast model that a weather app polls every minute. The serving layer has a latency target, written as an SLO (service level objective), for example "p99 latency below 100 ms". Two rates decide most of the design: the arrival rate $\lambda$ and the service rate $\mu$ of one replica, both in requests per second. Queueing theory turns them into latency. It explains a fact that surprises many teams: at 90% load the mean latency is already 10 service times. Serving systems such as NVIDIA Triton, TorchServe, KServe and vLLM all expose the same few settings (replicas, max batch size, max queue delay), and this lesson shows how to choose them. "Design a model-serving endpoint" is also a standard interview question.

1. Predict first #

2. The picture #

HTTP in one paragraph. A client sends a request: a method (GET to read, POST to send data), a path such as /predict, and often a JSON body such as {"t2m_k": 285.1, "rh": 71}. The server answers with a status code and a body. Codes 200–299 mean success, 400–499 mean "the client sent something wrong", and 500–599 mean "the server cannot do it now". The model server of this lesson is such a program.

One replica, one queue. Start with the simplest server: one replica that serves one request at a time, first in, first out (FIFO).

• A request arrives, waits in the queue, is served, and leaves. Its latency is the waiting time plus the service time.
• Throughput is the number of requests finished per second. A replica with mean service time $1/\mu$ finishes at most $\mu$ requests per second.
• The utilisation $\rho = \lambda/\mu$ is the share of time the replica is busy.

Why does a queue form when $\rho < 1$? Because arrivals are random. If requests came exactly every $1/\lambda$ seconds and took exactly $1/\mu$, no request would ever wait. Real requests come in bursts, and the replica clears the queue of a burst only with its spare time. At $\rho = 0.9$ there is little spare time, so the queue of one burst is often still there when the next burst comes.

Interactive figure: loading…

Now put the same numbers on one curve. The figure below shows the mean latency and the p99 latency of one replica against $\rho$. Both curves are flat at first and then rise without limit near $\rho = 1$: a wall.

Interactive figure: loading…

Batching, as a picture. A GPU pays a large fixed cost per model call (reading the weights, launching kernels) and a small cost per request. A batch of $b$ requests takes

The throughput $b / t(b)$ grows with $b$ towards $1/c$, but every request in a batch waits for the whole batch. So batching trades a little latency per call for a lot of throughput, and under load the extra throughput makes the queue much shorter. Section 3.4 makes this exact.

3. Definitions and main results #

3.1 The serving API #

Endpoint	Purpose	Answer
GET /healthz	liveness: the process runs and answers	200 always; never depends on the model
GET /readyz	readiness: may this replica get traffic?	200 when the model is loaded; 503 with a reason before
GET /version	which artifact answers	the model version (often also the code commit and the image digest)
POST /predict	one instance	200 with the prediction; 422 invalid input; 503 not ready
POST /predict_batch	a list of instances	one vectorised model call; 413 if the batch is too large

The status codes: 422 (unprocessable content) means "the body does not match the schema"; 413 (content too large) means "the batch is larger than the limit"; 503 (service unavailable) means "not ready, try another replica or later". Five rules make the server robust.

1. Load the model once, at startup. In FastAPI this is the lifespan handler: the code before yield runs before the first request, the code after it at shutdown. Not at import time (every test and tool that imports the module would load it), and not per request (seconds each).
2. Liveness never depends on the model (lesson 04, section 3.3). A liveness failure restarts the container; a readiness failure only removes the Pod from the Service. If loading takes 90 s and liveness checks the model, the container is restarted before loading finishes: a restart loop. For a slow load add a startup probe.
3. Validate the input with a schema: required fields, types, ranges, and no unknown fields. Invalid input then never reaches the model, and the client gets a 422 with the reason. This is the API's data contract (lesson 01): a temperature sent in °C to a model trained in kelvin is a valid number, and only a range check catches it.
4. One model call per batch, and a size limit (413), so that one client cannot take all the memory.
5. State lives in the app object (for example app.state in FastAPI), not in module-level global variables, so two apps in one process (two tests) stay independent.

A FastAPI sketch for the rain-nowcast model, as an illustration (other frameworks have the same parts; the notebook does not run web servers):

from contextlib import asynccontextmanager
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, ConfigDict, Field

class StationObs(BaseModel):                       # the input contract: FastAPI answers 422 on failure
    model_config = ConfigDict(extra="forbid")      # unknown fields are errors
    t2m_k: float = Field(gt=180, lt=340)           # 2 m temperature in kelvin
    rh: float = Field(ge=0, le=100)                # relative humidity in %

@asynccontextmanager
async def lifespan(app: FastAPI):
    app.state.model = load_model("nowcast-v3")     # once, before the first request (load_model: your loader)
    yield                                          # the app serves requests here

app = FastAPI(lifespan=lifespan)

@app.get("/healthz")
def healthz():
    return {"status": "ok"}                        # liveness: never touches the model

@app.post("/predict_batch")
def predict_batch(items: list[StationObs]):
    if len(items) > 64:
        raise HTTPException(status_code=413, detail="at most 64 instances")
    X = [[o.t2m_k, o.rh] for o in items]
    return {"p_rain": app.state.model.predict_proba(X)[:, 1].tolist()}   # one call for the batch

3.2 Latency, throughput and percentiles #

Latency is a distribution, not one number. Sort the $n$ measured latencies: $x_{(1)} \le x_{(2)} \le \dots \le x_{(n)}$. The $q$-th percentile (nearest-rank definition) is $x_{(\lceil q n / 100 \rceil)}$: the smallest value that at least $q\%$ of the requests do not exceed. The p50 is the median; the p99 is the latency that 99% of requests stay below. SLOs use high percentiles because the mean hides the slow requests.

Example. 97 requests take 12 ms and 3 take 400 ms ($n = 100$).

• Mean: $(97 \cdot 12 + 3 \cdot 400)/100 = (1164 + 1200)/100 = 23.6$ ms.
• p50: rank $\lceil 50 \rceil = 50$, so 12 ms. p95: rank 95, so 12 ms.
• p99: rank 99. The sorted list has 12 ms at ranks 1–97 and 400 ms at ranks 98–100, so 400 ms.

The mean (23.6 ms) describes no real request: almost every request is faster, and 3% are 17 times slower.

Percentiles cannot be averaged. Replica A serves 970 requests with p99 = 46 ms; replica B is slow and serves 30 requests with p99 = 636 ms (the "two replicas" sample in the figure below). The mean of the two p99 values is 341 ms, but the p99 of all 1,000 requests is 239 ms: the slow replica served only 3% of the traffic. The p99 over both depends on how many requests each replica served. Merge the histograms (or the raw latencies), then compute the percentile.

Result (fan-out amplifies the tail). A user request calls $n$ backends and waits for all of them. If each backend is slow with probability $q$, independently, the user request is slow with probability

Proof. The request is fast only if all $n$ backends are fast. By independence this has probability $(1-q)^n$. The complement is $1 - (1-q)^n$. $\blacksquare$

With $q = 1\%$: $1 - 0.99^{20} = 18.2\%$ for $n = 20$ and $1 - 0.99^{100} = 63.4\%$ for $n = 100$ (Dean and Barroso, "The Tail at Scale", 2013). For small $nq$, the value is close to $nq$.

Interactive figure: loading…

Little's law. Latency is a time per request; throughput is a rate. Little's law links them in any stable system (one that does not grow without bound):

with $L$ the mean number of requests in the system, $\lambda$ the arrival rate and $W$ the mean time a request spends in the system. It needs no assumption about the distributions. Example: a service gets 200 requests/s and has a mean latency of 50 ms, so on average $200 \cdot 0.05 = 10$ requests are inside it. This is how you size thread pools, connection pools and GPU batch slots.

Why Little's law holds (sketch)

Watch the system for a long time $T$ and draw $N(t)$, the number of requests inside at time $t$.

1. The area under $N(t)$ is $\int_0^T N(t)\,dt$. Each request adds 1 to $N(t)$ for exactly the time it is inside, so the area also equals the sum of the times in the system of all $A$ requests (up to the few requests still inside at the end, which matter less and less as $T$ grows).
2. Divide by $T$: $\frac{1}{T}\int_0^T N(t)\,dt = \frac{A}{T} \cdot \frac{1}{A}\sum_{i} W_i$.
3. As $T \to \infty$, the left side tends to $L$, $A/T$ to $\lambda$, and the mean of the $W_i$ to $W$. So $L = \lambda W$. $\blacksquare$

Sigman's notes on Little's law (see section 11) give the full sample-path proof.

3.3 The M/M/1 queue #

The notation M/M/1 means: arrivals form a Poisson process ("M" for memoryless, rate $\lambda$), service times are exponential ("M", rate $\mu$), and there is 1 server. Requests are served FIFO, and the queue has no size limit.

Theorem (M/M/1 mean latency). If $\rho = \lambda/\mu < 1$, then

The mean waiting time (before service) is $W_q = W - 1/\mu = \rho/(\mu - \lambda)$, and the mean number in the system is $L = \rho/(1 - \rho)$.

Why this is true, in words. In steady state the system must move up from $n$ to $n+1$ requests as often as it moves down from $n+1$ to $n$. This "balance" makes each extra request in the queue $\rho$ times as likely as one fewer: a geometric distribution. Its mean is $\rho/(1-\rho)$, and Little's law turns that into a time.

Proof.

1. Let $N(t)$ be the number of requests in the system. Arrivals and services are memoryless, so the future of $N(t)$ depends only on its present value: $N(t)$ is a birth–death Markov chain. It jumps $n \to n+1$ at rate $\lambda$ (an arrival), and $n \to n-1$ at rate $\mu$ for $n \ge 1$ (a service ends).
2. Let $\pi_n$ be the long-run share of time with $N = n$. Draw a cut between the states $n$ and $n+1$. The chain cannot cross this cut upwards twice without crossing it downwards in between, so in the long run it crosses up as often as down. Up-crossings happen at rate $\lambda\pi_n$ and down-crossings at rate $\mu\pi_{n+1}$, so $\lambda\pi_n = \mu\pi_{n+1}$.
3. Hence $\pi_{n+1} = \rho\,\pi_n$, and by induction $\pi_n = \rho^n \pi_0$.
4. The shares sum to 1: $\pi_0 \sum_{n \ge 0} \rho^n = \pi_0/(1-\rho) = 1$. The geometric series converges only for $\rho < 1$: this is the stability condition. So $\pi_n = (1-\rho)\rho^n$. The server is idle a share $\pi_0 = 1 - \rho$ of the time, as the word "utilisation" says.
5. The mean number in the system is $L = \sum_{n \ge 0} n(1-\rho)\rho^n = (1-\rho)\cdot\dfrac{\rho}{(1-\rho)^2} = \dfrac{\rho}{1-\rho}$, using $\sum_{n \ge 0} n\rho^n = \rho/(1-\rho)^2$.
6. Little's law $L = \lambda W$ gives $W = \dfrac{\rho}{\lambda(1-\rho)} = \dfrac{1}{\mu(1-\rho)} = \dfrac{1}{\mu - \lambda}$. $\blacksquare$

The tail. In the FIFO M/M/1 queue the latency $T$ of a request is itself exponential with rate $\mu - \lambda$: $P(T > s) = e^{-(\mu - \lambda)s}$. So its $q$-quantile is

Every percentile carries the factor $1/(1-\rho)$. It grows without limit as $\rho \to 1$, and faster the closer you get: from $\rho = 0.9$ to $\rho = 0.95$, every latency doubles, because $(1 - 0.9)/(1 - 0.95) = 2$.

Why the M/M/1 latency is exponential

1. An arriving request sees the system in its long-run state: a Poisson arrival finds $n$ requests with probability $\pi_n = (1-\rho)\rho^n$ (PASTA: Poisson arrivals see time averages).
2. It then stays for $n + 1$ service times: the rest of the current service (again exponential with rate $\mu$, by the memoryless property), $n - 1$ full services of the others, and its own. So $T$ is a sum of $K = n + 1$ independent exponential($\mu$) times, where $P(K = k) = (1-\rho)\rho^{k-1}$ for $k \ge 1$: a geometric number of terms.
3. A geometric($1-\rho$) sum of exponential($\mu$) times is exponential with rate $\mu(1-\rho)$. (Think of the service ends as a Poisson process with rate $\mu$; each end is "the last one" with probability $1 - \rho$, independently; thinning a Poisson process with probability $1-\rho$ gives a Poisson process with rate $\mu(1-\rho)$, and $T$ is the time to its first event.)
4. $\mu(1-\rho) = \mu - \lambda$. $\blacksquare$

Other service-time distributions: the Pollaczek–Khinchine formula

Model latency usually varies less than an exponential. For Poisson arrivals and any service-time distribution with mean $1/\mu$ (the M/G/1 queue), the mean waiting time is

$$W_q = \frac{\rho}{1-\rho} \cdot \frac{1 + C_s^2}{2} \cdot \frac{1}{\mu},$$

where $C_s$ is the coefficient of variation of the service time (standard deviation divided by mean). Exponential service has $C_s = 1$ and gives back $W_q = \rho/(\mu - \lambda)$. A constant service time ($C_s = 0$, the M/D/1 queue) halves the waiting time. But the factor $1/(1-\rho)$ stays for every $C_s$: no service distribution removes the wall. So M/M/1 is a slightly pessimistic planning model for the service times. Arrivals that are burstier than Poisson (for example many clients that poll on the hour) make the real queue worse.

3.4 Dynamic batching #

A dynamic batcher keeps a queue in front of the model. It sends a batch when the replica is free and either the queue holds max_batch_size $= B$ requests or the oldest request has waited the max queue delay. (Triton calls these max_batch_size and max_queue_delay_microseconds; TorchServe calls them batch_size and max_batch_delay.) The delay acts differently at three loads:

• Low load: the batch never fills; each request waits the full delay, then runs alone. The delay only adds latency.
• Medium load: the delay lets a few more requests join, so the throughput rises.
• High load: requests pile up while the replica is busy; batches form without waiting. The delay does not matter any more.

Result (the batch sizes itself under load). Suppose the replica is always busy and takes all waiting requests when it becomes free. Then the next batch holds the requests that arrived during the previous one: $b_{k+1} = \lambda\, t(b_k) = \lambda(a + c\,b_k)$. If $\lambda c < 1$, the batch size converges to the fixed point

Proof.

1. A fixed point satisfies $b = \lambda a + \lambda c\, b$, so $b(1 - \lambda c) = \lambda a$ and $b^* = \lambda a / (1 - \lambda c)$.
2. Subtract the fixed-point equation from the recursion: $b_{k+1} - b^* = \lambda c\,(b_k - b^*)$. So the distance to $b^*$ shrinks by the factor $\lambda c < 1$ at every batch, and $b_k \to b^*$. $\blacksquare$

If $b^* > B$, the batches are capped at $B$, and the replica's capacity is $B / t(B)$ requests per second. Above that rate the queue grows without bound. So $B$ is a cap for bursts, and the largest throughput of one replica is $B/t(B) < 1/c$.

Choosing the settings from an SLO.

1. Measure $t(b)$ for $b = 1, 2, 4, 8, \dots$ on the real hardware.
2. Pick $B$ so that the capacity $B/t(B)$ is well above the peak rate per replica (room for bursts) and $t(B)$ is well below the latency budget.
3. Pick a small max delay (a few ms, often below $t(1)$). At low load it costs exactly the delay.
4. Confirm the p99 at peak with a load test or a simulation. Add replicas instead of pushing one replica towards its capacity.

Interactive figure: loading…

Answers to section 1. (1) About 5. With $\mu = 50$/s, $W$ goes from $1/(50 - 25)$ s $= 40$ ms to $1/(50 - 45)$ s $= 200$ ms, while the traffic grows only 1.8 times (section 3.3). (2) Worse. The batch never fills, so each request waits the full 2 ms and then runs alone: about 12.5 ms instead of 10.5 ms (section 3.4). (3) About 18%: $1 - 0.99^{20} = 0.182$ (section 3.2).

4. Worked example #

Part A: replicas for a CPU model. A quality-control model for weather-station data runs on CPU replicas with a mean service time of 20 ms ($\mu = 50$/s). Treat each replica as M/M/1.

	$\lambda = 25$/s	$\lambda = 45$/s
$\rho = \lambda/\mu$	0.5	0.9
$W = 1/(\mu - \lambda)$	40 ms	200 ms
$L = \lambda W$	1	9
$W_q = W - 1/\mu$	20 ms	180 ms
$p_{99} = \ln 100/(\mu - \lambda)$	184 ms	921 ms

The SLO is p99 < 250 ms at a peak of 120 requests/s, spread evenly over the replicas.

1. p99 < 0.25 s needs $\ln(100)/(\mu - \lambda) \le 0.25$, so $\mu - \lambda \ge \ln(100)/0.25 = 18.4$/s.
2. So one replica may take at most $\lambda = 50 - 18.4 = 31.6$/s ($\rho \le 0.63$).
3. Replicas: $\lceil 120/31.6 \rceil = \lceil 3.8 \rceil = 4$.
4. Check: with 4 replicas each gets 30/s, $\rho = 0.6$, and $p_{99} = 4.605/20$ s $= 230$ ms ✓. With 3 replicas each gets 40/s, and $p_{99} = 4.605/10$ s $= 461$ ms ✗.

Interactive figure: loading…

Part B: batch settings for a GPU model. A precipitation-nowcasting model on one GPU needs $t(b) = 10 + 0.5\,b$ ms. The peak traffic is 400 requests/s, and the SLO is p99 < 50 ms.

1. Capacity $b/t(b)$: 95/s for $b = 1$, 571/s for $b = 8$, 889/s for $b = 16$, 1231/s for $b = 32$ (limit $1/c = 2000$/s). Without batching the replica is overloaded ($400 > 95$).
2. Self-sized batch at peak: $\lambda = 0.4$ per ms, so $b^* = 0.4 \cdot 10/(1 - 0.4 \cdot 0.5) = 4/0.8 = 5$ and $t(b^*) = 12.5$ ms. A request waits on average about half a batch time, then runs for one: about $1.5 \times 12.5 \approx 19$ ms.
3. Choice: $B = 16$ (capacity 889/s, more than twice the peak; a full batch takes 18 ms) and a max queue delay of 2 ms (its cost at night: about 2 ms per request).
4. A simulation of the batcher (Poisson arrivals, 200,000 requests) gives at 400/s: mean batch 5.0, mean latency 19.3 ms, p99 27.7 ms ✓. At 800/s the p99 is about 48–49 ms, just under the limit: time to add a replica. At 1000/s (above 889/s) the queue grows without bound. At 20/s the mean latency is 13.2 ms with the 2 ms delay and 11.6 ms without it.

5. In code #

Plain Python and NumPy check the schema rule, the sums of the proof, and the M/M/1 formulas against a simulated FIFO queue. The notebook has these cells and the exercises.

import math
import numpy as np

# 1. A plain-Python input contract: required fields, ranges, no extra fields
SCHEMA = {"t2m_k": (180.0, 340.0),    # 2 m temperature in kelvin
          "rh": (0.0, 100.0)}         # relative humidity in %

def validate(body, schema=SCHEMA):
    errors = [(k, "missing") for k in schema if k not in body]
    errors += [(k, "extra_forbidden") for k in body if k not in schema]
    for k, (lo, hi) in schema.items():
        if k in body and not lo <= body[k] <= hi:          # NaN fails every comparison
            errors.append((k, "out_of_range"))
    return errors                                          # an API answers 422 if not empty

for body in ({"t2m_k": 285.1, "rh": 71},
             {"t2m_k": 12.0, "rh": 71},               # sent in °C by mistake
             {"t2m_k": float("nan"), "rh": 71},       # NaN
             {"rh": 71},                              # missing field
             {"t2m_k": 285.1, "rh": 71, "wind": 3}):  # unknown field
    print(validate(body))
# []
# [('t2m_k', 'out_of_range')]
# [('t2m_k', 'out_of_range')]
# [('t2m_k', 'missing')]
# [('wind', 'extra_forbidden')]

# 2. Steps 4-5 of the proof at rho = 0.9: the pi_n sum to 1 and L = rho / (1 - rho) = 9
rho, n = 0.9, np.arange(2000)
print(round(((1 - rho) * rho**n).sum(), 6), round((n * (1 - rho) * rho**n).sum(), 6))   # 1.0 9.0

# 3. A FIFO queue against the M/M/1 formulas
def fifo_latencies(arrivals, services):
    """Latency of each request in a single-server FIFO queue."""
    lat, finish = np.empty_like(arrivals), -np.inf
    for i, (a, s) in enumerate(zip(arrivals, services)):
        finish = max(finish, a) + s       # start when both the request and the server are free
        lat[i] = finish - a
    return lat

rng, mu, N = np.random.default_rng(0), 50.0, 500_000
for lam in (25.0, 45.0):
    arr = np.cumsum(rng.exponential(1 / lam, N))
    lat = fifo_latencies(arr, rng.exponential(1 / mu, N))
    print(f"lambda={lam:.0f}/s: mean {1e3 * lat.mean():.0f} ms (theory {1e3 / (mu - lam):.0f}), "
          f"p99 {1e3 * np.percentile(lat, 99):.0f} ms (theory {1e3 * math.log(100) / (mu - lam):.0f})")
# lambda=25/s: mean 40 ms (theory 40), p99 185 ms (theory 184)
# lambda=45/s: mean 199 ms (theory 200), p99 943 ms (theory 921)

# 4. Constant service time (M/D/1) at lambda = 45/s: the mean wait halves, 180 ms -> 90 ms
wait = fifo_latencies(arr, np.full(N, 1 / mu)) - 1 / mu
print(f"M/D/1 mean wait {1e3 * wait.mean():.0f} ms (theory 90)")   # M/D/1 mean wait 89 ms (theory 90)

The simulated values land within a few percent of the theory. Near $\rho = 1$ one run is noisy, because the queue length changes slowly and one long burst can move the p99 of a whole run: at $\rho = 0.9$, runs of 200,000 requests with other seeds give p99 values 10–20% above or below the theory. Load tests near full load need long runs for the same reason. np.percentile interpolates between ranks by default; with many samples it agrees with the nearest-rank value.

6. Where it shows up #

• ML products. Ranking, fraud and search models are APIs with latency SLOs, often called with fan-out (one page calls many models). LLM servers such as vLLM do not wait for a batch to form; they use continuous batching: new requests join the running batch at each decoding step. The ML systems track covers this (ML systems lesson 12).
• ML for science. A nowcasting service gets bursts on the hour, when many clients poll at once: not Poisson, so plan for the peak. A surrogate called from a climate or ground-motion simulation runs inside the simulation, but the batch rule is the same: one vectorised call for all grid columns or scenarios. Moving inference from batch jobs to an always-on service is the step from throughput thinking to latency thinking.
• Practice. Kubernetes probes and autoscaling on request rate or latency (lesson 04); Triton, TorchServe and KServe expose a max batch size and a max delay; capacity plans use Little's law and a target utilisation well below 1.

7. Common mistakes #

• Liveness that checks the model or a database: restart loops during slow loads, and one dependency outage restarts every replica.
• Loading the model per request or at import time. Load it once, in the startup (lifespan) handler.
• CPU-heavy inference inside async def in FastAPI: it blocks the event loop for every request of that worker. Use a plain def endpoint (FastAPI runs it in a thread pool) or worker processes.
• A Python loop over the batch: no batching gain. Make one vectorised model call.
• Planning for the mean. At $\rho = 0.95$ the mean latency is 20 service times. Report the p99, and never average percentiles across replicas: merge the histograms.
• An unbounded queue. Near or above capacity it grows without limit, and every request in it will miss the SLO anyway. Bound the queue and reject early (429 or 503), so that clients retry elsewhere.
• A long max queue delay "for throughput" at low traffic: it only adds latency.

8. Check yourself #

1. A service handles 300 requests/s with a mean latency of 80 ms. How many requests are inside it on average?
   Answer

   By Little's law, $L = \lambda W = 300 \times 0.08 = 24$ requests. This holds for any stable system, without assumptions about the distributions.
2. A batch takes $t(b) = 5 + 2b$ ms. What is the maximum throughput with $B = 4$, with $B = 16$, and the limit for large $B$?
   Answer

   $4/(13 \text{ ms}) \approx 308$/s; $16/(37 \text{ ms}) \approx 432$/s; the limit is $1/c = 1/(2 \text{ ms}) = 500$/s.
3. A request fans out to 50 shards and waits for all of them. You want at most 5% of requests to be slow. How rare must slowness be for one shard?
   Answer

   $1 - (1-q)^{50} \le 0.05$ gives $q \le 1 - 0.95^{1/50} \approx 0.00103$: about 0.1% per shard, 50 times rarer than the target for the whole request.
4. The model needs 90 s to load. What do /healthz and /readyz return during that time, and what does Kubernetes do with each?
   Answer

   `/healthz` returns 200, so the kubelet does not restart the container. `/readyz` returns 503, so the Pod stays out of the Service's endpoints and gets no traffic. A startup probe with failureThreshold × periodSeconds above 90 s (for example 30 × 10 s) holds back the other probes during the load.
5. In part B of section 4 the traffic doubles to 800/s. Estimate the batch size and the mean latency with the fixed-point formula.
   Answer

   $b^* = 0.8 \cdot 10/(1 - 0.8 \cdot 0.5) = 8/0.6 \approx 13.3$, $t(b^*) = 10 + 0.5 \cdot 13.3 \approx 16.7$ ms, mean latency about $1.5 \times 16.7 \approx 25$ ms. The simulation gives a mean batch of 13.3 and a mean latency of about 28 ms (random bursts hit the cap $B = 16$), with a p99 of about 48–49 ms: at the SLO limit, so add a replica.
6. Why is $\rho = 0.95$ a bad operating point even if the p99 is still inside the SLO today?
   Answer

   The latency factor $1/(1-\rho)$ is 20 there, and it changes fast: 5% more traffic ($\rho \to 1$) makes the queue unstable, and a small burst or one slow replica moves the p99 far above the SLO. Plan for a target utilisation with margin, and let the autoscaler add replicas before $\rho$ gets close to 1.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. State Little's law and what each symbol means.

   Show answer

2. Give the M/M/1 mean latency.

   Show answer

3. In an M/M/1 queue, how does the p99 latency relate to the mean latency?

   Show answer

4. What is the utilisation of a replica?

   Show answer

5. A request waits for 20 backends, each slow 1% of the time, independently. Which share of requests is slow?

   Show answer

6. A batch of $b$ requests takes $t(b) = a + cb$. What is the throughput, and its limit?

   Show answer

7. What batch size does a busy dynamic batcher settle at?

   Show answer

8. What does a max queue delay cost at low traffic?

   Show answer

9. Why must a model server's liveness probe not check the model?

   Show answer

10. Why is an unbounded request queue a bug?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "Design the serving endpoint for this model." — Schema and 422; single and batch endpoints with a size limit; liveness, readiness and startup probes; a version endpoint; the model loaded once; dynamic batching on a GPU; replicas from the p99 SLO and a target utilisation; autoscaling; p99 monitoring from merged histograms.
• "The p99 doubled, the p50 did not move. Where do you look?" — Utilisation near 1, one slow replica, batches that are too large, garbage-collection pauses, cold starts, a slow dependency in a fan-out. Fixes: more replicas, a lower target utilisation, timeouts, hedged requests.
• "Why not run at 95% load?" — $\mu W = 1/(1 - \rho) = 20$, and small changes in traffic move the latency a lot.
• "Derive the mean latency of an M/M/1 queue." — Cut equations $\lambda \pi_n = \mu \pi_{n+1}$, geometric $\pi_n = (1-\rho)\rho^n$, $L = \rho/(1-\rho)$, Little's law.

10. Practice #

The notebook for this lesson runs in Colab. It has five exercises, each with a check cell and a folded solution:

1. latency_percentiles(samples, qs) with nearest rank; compare with np.percentile.
2. fifo_latencies(arrivals, services), then check $W$ and $p_{99}$ against M/M/1 at two loads, and the halving of the wait with constant service times.
3. simulate_batching(arrivals, a, c, max_batch, max_delay) with the dispatch rule of section 3.4; find the fixed point numerically and the load at which the p99 crosses the SLO.
4. replicas_for_slo(peak, mu, p99_slo) for M/M/1 replicas.
5. validate_request(body, schema) in plain Python: required fields, types, ranges and extra fields, with the list of errors.

11. Watch and read #

The M/M/1 QueueGareth Tribello · 20 min

A lecture on the M/M/1 queue; compare its results with the proof in section 3.3.

22. Sliding window analysis, Little's lawMIT OpenCourseWare · 6.02, Hari Balakrishnan · 53 min

Little's law starts at about minute 36, with the area argument used in the proof sketch above.

Lecture 05: Deployment (FSDL 2022)The Full Stack · Full Stack Deep Learning 2022 · 57 min

From 14:03: REST APIs for model services, containers, GPU batching (29:58) and horizontal scaling.

• Karl Sigman, Notes on Little's Law (the law, its sample-path proof, and the M/M/1 queue), and Continuous-Time Markov Chains (birth–death processes and the balance equations).
• J. Dean and L. A. Barroso, "The Tail at Scale", Communications of the ACM 56(2), 2013: fan-out, tail latency and hedged requests.
• NVIDIA Triton, Batchers: "Dynamic Batcher" and "Recommended Configuration Process".
• FastAPI, Lifespan Events: loading a model once at startup.