ML system design: the framework, recommendation, search and ads

Before this lesson: lesson 09 (precision, recall, AUC, calibration and the A/B test sample size). Lesson 12 and lesson 13 give more back-of-envelope practice (Little's law, latency percentiles); they are useful but not required. For depth: the in-batch softmax loss is the InfoNCE loss of deep learning lesson 14; PSI is built from the KL divergence of information theory lesson 3; deployment and monitoring are in MLOps lesson 6, MLOps lesson 7 and MLOps lesson 8. Time: about 110 minutes, plus the notebook and one timed mock design.

Why this matters #

ML system design is a standard round for ML engineers and research engineers, and some research scientist loops include it too. The prompt is short: "Design a feed", "Design product search", "Design ad click prediction". The interviewer wants to see a complete system with numbers, not a favourite model.

Strong researchers often fail this round in one typical way: they spend 30 minutes on the model architecture and never reach metrics, serving numbers or monitoring. A fixed framework and a few back-of-envelope formulas prevent this.

The same skills matter in ML for science. Before a forecast model runs every day, someone must define the target metric, the data pipeline, the time budget of one forecast cycle, the evaluation against baselines, and the monitor that notices when an input instrument changes.

1. Predict first #

2. The picture #

2.1 The funnel #

Start with the simplest case. A shop sells 1,000 products. For each visitor, one model scores all 1,000 products and the page shows the top 10. One model and one stage are enough.

Now grow the catalog to 50 M videos and the traffic to thousands of requests per second. Scoring every item with a rich model costs more than any data center has (question 1). The solution is a funnel: each stage sees fewer items and can spend more compute on each one.

• Retrieval (also called candidate generation) picks a few hundred candidates from the whole catalog. It must not miss good items, so it optimizes recall: the share of good items that it finds. It is cheap per item, because the item side is computed in advance.
• Ranking orders the few hundred candidates. It must put the best items first, so it optimizes precision at the top. It can afford expensive features, such as the user's past interactions with this exact creator.
• Re-ranking applies what a score per item cannot express: diversity across the page, freshness quotas, business and policy rules.

2.2 The framework is a funnel for your 45 minutes #

The same idea applies to your time. First come the cheap parts with high value: the goal, the metrics, the numbers and a complete sketch. Then come the details, where the interviewer wants depth. A design that is complete and shallow at minute 27 can still become deep. A design that is deep in one corner at minute 40 cannot become complete.

3. Definitions and main results #

3.1 The 45-minute framework #

Three rules make the framework work:

1. Numbers before boxes. State assumptions as numbers ("say 100 M daily users, 5 page loads each"). Then let the numbers choose the design.
2. Breadth first, then depth. Reach a complete design by about minute 27. Then go deep where the interviewer points.
3. Every choice with its alternative. "A neural ranker, because it learns from id embeddings and the watch history; gradient-boosted decision trees (GBDT) are the baseline."

This time plan is close to the public "delivery framework" of Hello Interview and to the seven-step framework of Xu and Aminian (see section 11).

3.2 Metrics at four levels #

Here p99 latency is the 99th percentile: 99% of requests are faster. Compute every offline metric on a time period after the training data. The definitions:

• Recall@K (retrieval) = (number of relevant items among the $K$ candidates) / (number of relevant items), averaged over users. The relevant items are those the user engaged with in the evaluation period.
• NDCG@k (normalized discounted cumulative gain; ranking with graded labels). Let $\text{rel}_i$ be the label of the item at position $i$. Then

where IDCG@k (the ideal DCG) is the DCG of the best possible order. NDCG is in $[0, 1]$. The discount $1/\log_2(i+1)$ rewards good items near the top. Some libraries (scikit-learn among them) use the linear gain $\text{rel}_i$ instead of $2^{\text{rel}_i} - 1$; say which one you use.

• MRR (mean reciprocal rank; search) = the mean over queries of $1/(\text{rank of the first relevant result})$.
• AUC = the probability that a random positive gets a higher score than a random negative. It ignores calibration. A page only orders the items of one request, so the per-user AUC (averaged over users) is closer to what matters than one global AUC.
• Normalized entropy (NE) (He et al., 2014): the average log loss divided by the entropy of the background rate $p$ (the mean label):

NE $< 1$ means better than always predicting $p$; lower is better. Example: $p = 0.05$ gives a denominator of 0.1985 nats, so a model with log loss 0.170 has NE = 0.856.

• Calibration ratio = (sum of predicted probabilities) / (number of observed positives). The target is 1. Ads need it (section 3.9).

3.3 Back-of-envelope formulas #

Write $U$ for the daily active users, $r$ for the requests per user per day, and $f$ for the peak factor (peak traffic divided by average traffic).

• Queries per second (QPS). Average QPS $= U r / 86{,}400$ (a day has 86,400 seconds); peak QPS $= f \times$ average. A day is about $10^5$ s, so $10^8$ requests per day are about 1,160 QPS.
• Index memory $= N d b$ for $N$ items, embedding dimension $d$ and $b$ bytes per number (4 for float32, 2 for float16). An HNSW index (hierarchical navigable small world: a layered graph of near neighbours) adds about $2M$ neighbour ids of 4 bytes per vector, where $M$ is the graph degree. Product quantization (PQ) splits each vector into $m$ sub-vectors and stores each one as a 1-byte cluster id (256 centroids per sub-space): $m$ bytes per vector instead of $d b$. IVF-PQ first assigns each vector to a coarse cluster (an inverted file) and searches only a few clusters.
• Ranker load $=$ peak QPS $\times K$ scorings per second, for $K$ candidates per request. Ranker compute $\approx$ load $\times 2P$ FLOP/s for a model with $P$ dense parameters (one multiply and one add per parameter).
• Log volume $=$ requests per day $\times$ items shown per request $\times$ bytes per logged row.

For GPU counts we assume one GPU sustains 30 TFLOP/s on a ranking model. This is about 3% of the dense bf16 peak of an H100 (989 TFLOP/s): small batched MLPs and embedding lookups are limited by memory traffic, not by arithmetic. State such an assumption aloud in the interview.

3.4 Retrieval: several sources and a two-tower model #

Retrieval runs several sources in parallel and merges them: a personalized two-tower model, item-to-item co-engagement ("users who watched your last video also watched …"), subscriptions, and popular or fresh items per language and region. Several sources make the system robust and help with cold start. Each source returns 100–300 items. The merge removes duplicates and filtered items (already watched, blocked, against policy).

Two-tower model. A user tower $u_\theta(x) \in \mathbb{R}^k$ embeds the user and the context $x$ (watch history, searches, country, device, time). An item tower $v_\phi(y) \in \mathbb{R}^k$ embeds the item $y$ (an id embedding plus content features). Here $\theta, \phi$ are the weights of the towers and $k$ is the embedding dimension. The score is a dot product of normalized embeddings with a temperature $\tau > 0$:

The cosine is in $[-1, 1]$, so the logits are in $[-1/\tau, 1/\tau]$: a small $\tau$ allows sharp softmax distributions, a large $\tau$ forces flat ones. The score factorizes into a user part and an item part. So all item vectors are computed offline and stored in an approximate nearest neighbour (ANN) index, such as HNSW or IVF-PQ. At request time only the user tower runs, and the ANN search returns the top items in a few milliseconds. The price: the model cannot use cross features, that is, features of the (user, item) pair, such as "how often this user watched this creator". Cross features are the ranker's job.

Training with in-batch negatives. A batch holds $B$ positive pairs $(x_i, y_i)$: a user and an item the user engaged with. For user $i$, the other $B - 1$ items of the batch act as negatives:

This is the InfoNCE (CLIP) loss with one correction, $-\log q_j$. Here $q_j$ is the probability that a random pair from the training stream has item $j$. It grows with popularity, and it is estimated from the stream (Yi et al., 2019, use a streaming frequency estimate).

Result 1 (log-Q correction; Yi et al., 2019, §3). Without the correction, the in-batch softmax punishes popular items too much, because they are negatives in many batches. With the plain logits $s$, the learned score tends to $\log p(y \mid x) - \log q_y$ (plus a constant per user), so popular items rank too low. Subtracting $\log q_j$ from every logit removes this sampling bias.

Why (importance sampling).

1. The full softmax needs $Z(x) = \sum_{j \in \text{catalog}} e^{s(x, j)}$, a sum over 50 M items. That is too expensive for every training example.
2. The in-batch items are draws from the training stream: item $j$ comes with probability $q_j$.
3. For $B$ independent draws $j_1, \dots, j_B$: $\mathbb{E}\big[\tfrac{1}{B}\sum_{b} e^{s(x, j_b)}/q_{j_b}\big] = \sum_j q_j \, e^{s(x,j)}/q_j = Z(x)$. So $\tfrac{1}{B}\sum_b e^{s(x, j_b) - \log q_{j_b}}$ is an unbiased estimate of $Z(x)$: the corrected logits give the right normalizer.
4. The plain batch sum $\sum_b e^{s(x, j_b)}$ estimates $B \sum_j q_j\, e^{s(x,j)}$ instead: a popularity-weighted sum. Training then makes $s$ play the role of the corrected logit, so $s(x, y) \approx \log p(y \mid x) - \log q_y + \text{const}$.
5. Constant factors such as $1/B$ shift every logit by the same amount and cancel inside the softmax. $\blacksquare$

ANN in numbers. Exact search costs $N d$ multiply-adds per query: $5 \times 10^7 \times 128 = 6.4 \times 10^9$. At 11,600 QPS and 2 FLOPs per multiply-add, this is about $1.5 \times 10^{14}$ FLOP/s only to find candidates. An ANN index visits a small fraction of the vectors and, at a tuned setting, returns most of the exact top $K$ in a few milliseconds on CPUs.

3.5 Ranking: labels, negatives and the multi-task model #

Labels and negatives. The ranker learns from impressions: an item shown to a user, with what happened next. Positives are engagements. Negatives are impressions without the engagement. Do not use random catalog items as ranking negatives: at serving time the ranker must separate good candidates from almost-good ones, and only impressions look like that.

A click alone is a poor label, because it rewards clickbait: titles that attract clicks and are then skipped. Use one multi-task model with several heads on shared layers (a shared bottom, or a multi-gate mixture of experts): $P(\text{click})$, expected watch time, $P(\text{like})$ and $P(\text{dismiss})$. The final score is a fixed formula, for example

with weights $a, b \ge 0$ tuned by A/B tests.

Result 2 (weighted logistic regression predicts expected watch time; Covington et al., 2016, §4.2). Train logistic regression with each clicked impression weighted by its watch time $T_i$, and each non-clicked impression weighted by 1. Then the learned odds $e^{w^\top x}$ approximate the expected watch time per impression $\mathbb{E}[T]$, where $T = 0$ without a click.

Proof (for one cell of identical feature vectors $x$, so the model is one logit $z$).

1. Take $N$ impressions in the cell; $k$ of them are clicked, with watch times $T_1, \dots, T_k$. The weighted log loss is $\ell(z) = -\sum_{i=1}^{k} T_i \log\sigma(z) - (N-k)\log(1-\sigma(z))$, with $\sigma(z) = 1/(1+e^{-z})$.
2. Use $\frac{d}{dz}\log\sigma(z) = 1-\sigma(z)$ and $\frac{d}{dz}\log(1-\sigma(z)) = -\sigma(z)$: $\ell'(z) = -(1-\sigma)\sum_i T_i + (N-k)\,\sigma$.
3. Set $\ell'(z) = 0$: $\;e^{z} = \frac{\sigma}{1-\sigma} = \frac{\sum_i T_i}{N-k}$.
4. With the click rate $P = k/N$ and $\mathbb{E}[T] = \frac{1}{N}\sum_i T_i$: $e^z = \frac{\mathbb{E}[T]}{1-P} = \mathbb{E}[T]\,(1 + P + P^2 + \dots) \approx \mathbb{E}[T]\,(1+P)$.
5. Click rates on a home page are small (a few percent), so $e^z \approx \mathbb{E}[T]$. At serving time, use $e^{w^\top x}$ as the predicted watch time. $\blacksquare$

The identity of step 3 is exact when the model has one free parameter per cell, as with one-hot video ids (section 5 checks it). A model with rich features applies the same idea across cells.

Features. Four groups, each with a reason:

• user: watch history (the last 50 video ids as embeddings, averaged or attended), language, country, account age — long-term taste;
• item: video id embedding, content embedding (title, thumbnail, audio, from a pretrained encoder), duration, upload age, creator id, engagement rates over the last hour, day and week — quality and freshness;
• user × item (cross): the user's past watches of this creator, time since the user last watched this topic, how often this video was already shown to this user without a click — often the strongest features;
• context: time of day, day of week, device, network — the same user wants short videos on a phone at 8 am and long ones on a TV at 9 pm.

Ids become learned embeddings (they hold most of the parameters). Counts get a log or quantile transform. Rates are smoothed with a prior, because a video with few impressions has a noisy rate. The display position is a training-only feature (section 3.7).

Split by time. Train on days 1–28, validate on day 29, test on day 30. A random split leaks the future (the same trending video appears on both sides) and makes offline gains look larger.

3.6 Online evaluation #

An offline gain is a hypothesis; the A/B test decides. Covington et al. (§2) report that offline metrics and live A/B results do not always agree. Randomize by user, not by request: each user then sees one system, and the per-user metrics of the two arms are independent. For a per-user metric with standard deviation $\sigma$ and a minimum detectable effect $\delta$ (absolute), the sample size per arm at two-sided level $\alpha$ and power $1 - \beta$ is (lesson 09)

where $z_q$ is the $q$-quantile of the standard normal. For $\alpha = 0.05$ and power 0.8, $(z_{0.975} + z_{0.8})^2 = (1.960 + 0.842)^2 = 7.85$. Run full weeks (at least two) to cover weekly cycles and novelty effects. Check for sample-ratio mismatch, and fix the decision rule before the start.

Two tools that ranking teams use:

• CUPED (controlled experiments using pre-experiment data). Subtract the part of the metric that the same user's pre-experiment value predicts. With correlation $\rho$ between the two periods, the variance, and so $n$, falls by the factor $1 - \rho^2$.
• Interleaving. Merge the lists of the two rankers into one page per user, and count which ranker's items get the engagements. Each user compares both rankers, so this is much more sensitive than an A/B test.

3.7 Deep dives you must be ready for #

• Position bias. Items at the top get more clicks, whatever their quality, because users look at the top more often. A model trained on raw clicks learns "what the old system showed at the top". In the position-based click model, $P(\text{click} \mid \text{item } i \text{ at position } k) = \theta_k\, r_i$: the user examines position $k$ with probability $\theta_k$ (the propensity) and clicks an examined item with probability $r_i$ (its relevance). Fixes:
  • add the position as a training feature and set it to one fixed value at serving time;
  • or weight each click by the inverse propensity $1/\theta_k$ (IPS, inverse propensity scoring). Then $\mathbb{E}[\text{click}/\theta_k] = \theta_k r_i/\theta_k = r_i$ at every position, so the weighted click rate estimates the relevance without bias. The propensities are estimated from a small slice of traffic in which the order is randomized.

• Feedback loops. The model gets labels only for what it showed, so it reinforces its own past choices: popular items get more popular. Fixes: a small exploration budget (for example 1–5% of the slots for less-shown items), logged propensities for unbiased offline evaluation, and diversity rules.
• Cold start. A new video: the item tower uses content features, so a new video gets an embedding at upload time; a freshness pool and exploration slots collect its first signals. A new user: onboarding topics, popular items in the user's language and region, and context features (device, country, time).
• Training–serving skew. A feature computed one way offline and another way online breaks the model without any error message. Fixes: log the features at serving time and train on these logs; use one feature definition for both paths (a feature store with point-in-time correct joins, so a training row sees only the feature values that existed at that time).
• Delayed labels. Some labels arrive late: a purchase days after the click, a subscription cancelled weeks later. Training on recent data then sees too few positives. Fixes: wait a fixed attribution window before a row becomes a negative, or model the delay; for monitoring, use fast proxies (clicks, add-to-cart) and check them against the slow label later.
• Freshness. Behaviour changes every day. He et al. (2014, §3.2) found that training daily instead of weekly lowered NE by about 1%. Retrain the ranker daily or continuously, and add new uploads to the index within minutes.

3.8 Deployment and monitoring #

From a trained model to users. A new model passes through these steps, in this order:

1. Promotion gate. A written rule, checked by code on a frozen test set, candidate against production. For example: the overall metric drops by at most 0.5 points; no slice with at least $n_{\min}$ examples drops by more than 2 points; latency at the expected peak load stays within the target; the behavioural tests pass. A model registry stores the versions with their metrics and lineage, and an alias (champion) points to the version in production.
2. Shadow deployment. The new model receives a copy of the real traffic, but its answers are not shown to users. You compare its predictions, latency and errors with production. It sees real traffic with no user impact.
3. Canary release. A small share of users (for example 1%, then 5%, then 25%) gets the new model. Guardrails (errors, latency, the online metric) are watched at each step, and the rollout stops automatically if one of them gets worse.
4. A/B test for the decision on the product metric (section 3.6).
5. Rollback. Move the registry alias back to the previous version. Keep the previous model loaded or ready, so a rollback takes minutes, not a new training run.

What to monitor. Four layers:

• system: QPS, p50 and p99 latency per stage, errors, timeouts per retrieval source;
• data: feature null rates, freshness of streaming features, drift of the input distributions, the share of new items in the index;
• model: prediction drift, calibration ratio, NE on yesterday's data (when labels arrive), the share of each retrieval source in the final page;
• business: the online metric and the guardrails.

Drift statistics. Input drift (also called covariate shift) and prediction drift need no labels, so they are the earliest signals. The two standard statistics:

• PSI (population stability index). Bin the feature with the quantiles of the reference (training) data into $k$ bins. Let $p_i$ be the reference share and $q_i$ the live share in bin $i$. Then $\text{PSI} = \sum_i (q_i - p_i)\ln(q_i/p_i)$. A common rule of thumb (from credit scoring): below 0.1 stable, 0.1–0.25 moderate shift, above 0.25 large shift.
• KS (two-sample Kolmogorov–Smirnov): $D = \max_x |F_{\text{ref}}(x) - F_{\text{live}}(x)|$ over the empirical cumulative distribution functions, with a p-value. With thousands of samples, tiny and harmless shifts become "significant". Use $D$ as an effect size, or PSI with a threshold, and the p-value only as one input.

Result 3 (PSI is a symmetrized KL divergence). $\text{PSI} = \text{KL}(q\,\|\,p) + \text{KL}(p\,\|\,q)$.

Proof. $\text{KL}(q\|p) = \sum_i q_i \ln(q_i/p_i)$ and $\text{KL}(p\|q) = \sum_i p_i \ln(p_i/q_i) = -\sum_i p_i \ln(q_i/p_i)$. Add the two: $\sum_i (q_i - p_i)\ln(q_i/p_i)$. $\blacksquare$

So PSI $\ge 0$, it is 0 only when $p = q$, and it is symmetric. Each term is $\ge 0$, because $q_i - p_i$ and $\ln(q_i/p_i)$ have the same sign. An empty bin makes it infinite, so implementations floor the shares at a small $\varepsilon$.

Result 4 (the noise floor of PSI). With no drift, a reference of $n$ samples and a live window of $m$ samples give, for large $n$ and $m$,

Proof.

1. Write $q_i = p_i + d_i$ with a small $d_i$. Then $\ln(q_i/p_i) \approx d_i/p_i$, so each term $(q_i - p_i)\ln(q_i/p_i) \approx d_i^2/p_i$ (higher-order terms dropped).
2. So $\text{PSI} \approx \sum_i d_i^2/p_i$, the chi-square divergence.
3. With no drift, both shares estimate the same probability $\pi_i$ from independent samples, with variances $\pi_i(1-\pi_i)/n$ and $\pi_i(1-\pi_i)/m$. So $\mathbb{E}[d_i^2] \approx \pi_i(1-\pi_i)(1/n + 1/m)$.
4. Divide by $p_i \approx \pi_i$ and sum: $\sum_i (1-\pi_i)(1/n + 1/m) = (k-1)(1/n + 1/m)$, because $\sum_i \pi_i = 1$. $\blacksquare$

With $k = 10$ bins and $n = m = 200$, the floor is already $9 \times (1/200 + 1/200) = 0.09$, close to the 0.1 threshold, with no drift at all. A simulation shows that 37% of such windows exceed 0.1. So choose the window so that the floor is far below the alert threshold, and alert with hysteresis: raise the alert above 0.25, and clear it only below 0.1. Then one noisy window does not switch the alert on and off.

project/
  pyproject.toml, uv.lock        pinned environment
  configs/*.yaml                 every hyperparameter (a tiny config for CI)
  data/raw/*.dvc                 pointers to versioned data (the data is not in git)
  src/proj/  data.py  train.py  evaluate.py  gate.py  serve.py  monitor.py
  tests/unit  tests/data  tests/model  tests/api
  Dockerfile, .github/workflows/ci.yml, README.md (diagram, numbers, how to run)

3.9 Search and ads: what changes #

Here pCTR and pCVR are the predicted click-through rate and conversion rate. Budget pacing spreads an advertiser's daily budget over the day, so it is not spent in the first hours.

In ads the predicted probability sets a price, so it must be calibrated, not only well ordered. Click rates are often below 1%, so teams keep all positives and only a fraction $w$ of the negatives (negative downsampling). This changes the base rate, and the predictions must be corrected.

Result 5 (recalibration after negative downsampling; He et al., 2014, §6.3). Assume the training data keep every positive and keep each negative with probability $w$, independently of the features. Let $p_s$ be a calibrated prediction on these sampled data. The probability on real traffic is

Proof.

1. Downsampling keeps every positive and a fraction $w$ of the negatives, independently of the features $x$. So for each $x$, the odds in the sampled data are $\frac{p_s}{1-p_s} = \frac{p}{(1-p)\,w}$.
2. Solve for the true odds: $\frac{p}{1-p} = w\,\frac{p_s}{1-p_s}$.
3. Turn the odds into a probability: $p = \frac{w\,p_s}{w\,p_s + 1 - p_s} = \frac{p_s}{p_s + (1-p_s)/w}$. $\blacksquare$

Example: a true click rate of 0.1% with $w = 0.01$ becomes about 9.1% in the sampled data; the formula maps 9.1% back to 0.1%.

Answers to section 1. (1) Scoring every video costs $11{,}574 \times (5 \times 10^7) \times (2 \times 10^7) \approx 1.2 \times 10^{19}$ FLOP/s: about 386,000 GPUs at 30 TFLOP/s each. Retrieval cuts the scored set to about 500 items per request, a factor of $10^5$, and the ranker then needs about 4 GPUs. (2) Possible reasons: the label (click) is not the objective (watch time), so a better click model can promote clickbait; the logged data come from the old system (position bias, feedback loop), so a gain on logged data need not transfer; training–serving skew (a feature has other values online); the ranker only reorders the same 500 candidates, so retrieval limits the gain; the test has too little power for the true effect.

4. Worked example: home-page video recommendations #

Prompt: "Design the recommendations on the home page of a video platform." We follow the time plan of section 3.1. All numbers are an invented but realistic example.

Clarify (0–5 min). You ask; the interviewer answers; you write the answers down.

Frame and metrics (5–8 min). Business objective: users find videos they value and come back. Online metric: watch time per user per day; secondary: 7-day return rate. Guardrails: p99 latency, dismiss rate, creator diversity (distinct creators in the top 10), crash rate. ML task: for a (user, context) pair, retrieve about 500 candidates from 50 M; for each candidate, predict $P(\text{click})$, expected watch time, $P(\text{like})$ and $P(\text{dismiss})$; order by the score of section 3.5. Offline metrics: recall@500 for retrieval; AUC and NE per head, and NDCG@20, for ranking.

Numbers (8–11 min).

• Requests: $10^8 \times 5 = 5 \times 10^8$ per day, so $5 \times 10^8 / 86{,}400 = 5{,}787$ QPS on average and 11,574 QPS at peak.
• Retrieval index with $d = 128$: $5 \times 10^7 \times 128 \times 4$ B $= 25.6$ GB in float32, 12.8 GB in float16, 38.4 GB as HNSW ($M = 32$: $2 \times 32 \times 4 = 256$ extra bytes per vector, float32), 1.6 GB with 32-byte PQ codes. Each option fits in the memory of one server: replicate the index for throughput, no sharding.
• Ranker: $11{,}574 \times 500 = 5.79$ M scorings per second at peak. With $P = 10^7$ dense parameters: $5.79 \times 10^6 \times 2 \times 10^7 = 1.16 \times 10^{14}$ FLOP/s, about 4 GPUs at 30 TFLOP/s. Provision about twice that per region, for headroom and failures.
• Embedding tables dominate memory: the video ids alone take $5 \times 10^7 \times 64 \times 4$ B $= 12.8$ GB (64-dimensional, float32).
• Logs: $5 \times 10^8 \times 20 = 10^{10}$ impressions per day. With a 5% click rate, keeping all clicks and 10% of the non-clicks gives $5 \times 10^8 + 0.1 \times 9.5 \times 10^9 = 1.45 \times 10^9$ training rows per day, about 0.7 TB at 500 bytes per row. Training compute is $6P \times \text{rows} = 8.7 \times 10^{16}$ FLOPs per day: under one GPU-hour at 30 TFLOP/s. The cost of training a recommender lies in data loading and embedding lookups, not in FLOPs.

Latency budget (p99 ≤ 200 ms).

The slack matters because parallel calls make the tail worse. If a request waits for 10 calls, and each call is slower than its own p99 in 1% of the cases (independently), then at least one call is slow with probability $1 - 0.99^{10} = 9.6\%$. Give each source a timeout (answer without a slow source), and send a duplicate (hedged) request to a second replica when the first one is late.

Data and labels (11–19 min).

• Retrieval: positives are meaningful watches (for example, at least half of the video or at least 30 s). The input is the history before the watch, and the label is the next watch; predicting a randomly held-out watch leaks future information (Covington §3.4). Negatives: in-batch, with the log-Q correction. Use a fixed number of examples per user, so that heavy users do not dominate the loss.
• Ranking: one row per impression, with the features logged at serving time. One label per head: click, watch time, like, dismiss. Sampling: keep every clicked impression and 10% of the non-clicked ones.
• Features: the four groups of section 3.5, and the position for training only.

Model and training (19–27 min).

• Baselines: for retrieval, popular videos per country plus co-watch; for ranking, a GBDT on about 20 features that predicts $P(\text{click})$. Every later model must beat these online.
• Retrieval: a two-tower model; the user tower reads the last 50 watches and the context; the item tower reads the id and the content embedding; in-batch softmax with the log-Q correction; $k = 128$. Retrain weekly. Embed new uploads at upload time and add them to the index every 15 minutes.
• Ranking: a multi-task neural network (shared embeddings, a mixture-of-experts bottom, four heads). Binary cross-entropy for click, like and dismiss; weighted logistic regression for watch time (Result 2). The training rows keep only 10% of the non-clicks ($w = 0.1$), so correct the heads before they are combined. Click head: Result 5. Watch-time head: multiply the learned odds by $w$ (its weight-1 rows are exactly the non-clicks). Like and dismiss heads: Result 5 does not apply, because their negatives include every clicked row but only 10% of the non-clicked rows, and most dismissals are non-clicks. Give each kept non-click row the weight $1/w = 10$ in these two losses instead; this restores the true data distribution. Retrain daily, starting from yesterday's weights.
• Re-ranking: at most 2 videos per creator in the top 10; 2 of the 20 slots for fresh or exploration videos; policy filters.
• Split by time: train on 28 days, validate on day 29, test on day 30.

Evaluation (27–34 min).

• Offline retrieval, on day 30: a user watched 4 videos that day, and 3 of them are among the 500 candidates, so recall@500 = 3/4 = 0.75 for this user. Average over users.
• Offline ranking: graded labels 0 (shown, not clicked), 1 (clicked, short watch), 2 (watched more than half), 3 (liked). A page in model order has the labels $(2, 0, 3, 0, 1)$:

The best order $(3, 2, 1, 0, 0)$ gives $\text{IDCG@5} = \frac{7}{1} + \frac{3}{\log_2 3} + \frac{1}{\log_2 4} = 7 + 1.893 + 0.5 = 9.393$, so NDCG@5 $= 6.887 / 9.393 = 0.733$.

• Online: an A/B test with the user as the unit. In past data, the per-user average daily watch time over two weeks has mean 40 min and $\sigma = 60$ min (a heavy tail). The team wants to detect +0.5%, that is, $\delta = 0.2$ min:

1.4% of the daily users per arm. Run 14 days for weekly cycles and novelty. With CUPED and a pre-period correlation $\rho = 0.7$: $n \times (1 - 0.49) \approx 0.72$ M per arm. Ship if the primary metric improves at $p < 0.05$ and no guardrail gets worse.

Deployment, monitoring, trade-offs (34–45 min). The new ranker passes the promotion gate, runs in shadow for two days (compare score distributions and latency), then goes to a 1% canary and then to the A/B test. Dashboards on the four layers of section 3.8, with alerts: p99 latency above 200 ms; any retrieval source timing out in more than 1% of requests; a feature null rate that doubles; a calibration ratio outside $[0.9, 1.1]$; PSI of key features above 0.25 on windows of at least 10,000 requests (noise floor $9 \times (10^{-5} + 10^{-4}) \approx 0.001$ with $n = 10^5$ reference rows and 10 bins); a drop in the share of new videos in the index (a broken embedding job removes fresh candidates without any error). Then the deep dives of section 3.7 that the interviewer picks. Trade-offs to state: the speed of the two-tower model against the power of cross features; the cost of daily retraining against freshness; short-term engagement against long-term satisfaction (survey answers as a label for the second).

5. In code #

The numbers of section 4, NDCG with scikit-learn, Result 2 on simulated logs, the A/B sample size and the PSI noise floor:

import numpy as np
from scipy.stats import norm
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import ndcg_score

# (a) Back-of-envelope numbers of section 4
DAU, loads, peak_factor = 100e6, 5, 2
N_items, d, K, slate, P_dense = 50e6, 128, 500, 20, 10e6
qps_peak = peak_factor * DAU * loads / 86_400
print(f"peak QPS {qps_peak:,.0f}")                                   # 11,574
print(f"index fp32 {N_items * d * 4 / 1e9:.1f} GB, PQ-32 {N_items * 32 / 1e9:.1f} GB")
ranker = qps_peak * K * 2 * P_dense                               # FLOP/s of the ranker
print(f"ranker {ranker:.2e} FLOP/s = {ranker / 30e12:.1f} GPUs at 30 TFLOP/s")
print(f"rank all items: {qps_peak * N_items * 2 * P_dense / 30e12:,.0f} GPUs")
# index fp32 25.6 GB, PQ-32 1.6 GB
# ranker 1.16e+14 FLOP/s = 3.9 GPUs at 30 TFLOP/s
# rank all items: 385,802 GPUs

# (b) NDCG@5; scikit-learn uses the label itself as the gain, so pass 2**rel - 1
rel = np.array([[2, 0, 3, 0, 1]])                  # labels in the model's order
score = np.array([[5, 4, 3, 2, 1]])                # higher score = earlier position
print(round(ndcg_score(2**rel - 1, score, k=5), 4), round(ndcg_score(rel, score, k=5), 4))
# 0.7332 0.8162   (exponential gain, linear gain: two different numbers)

# (c) Weighted logistic regression learns odds = sum(T) / (N - k), close to E[T]
rng = np.random.default_rng(0)
n = 200_000
videos = {"A (long video)": (0.04, 8.0), "B (clickbait)": (0.10, 2.0)}   # click rate, minutes per click
ids, y, w = [], [], []
for g, (p_click, minutes) in enumerate(videos.values()):
    clicked = rng.random(n) < p_click
    watch = np.where(clicked, rng.exponential(minutes, n), 0.0)
    ids.append(np.full(n, g)); y.append(clicked); w.append(np.where(clicked, watch, 1.0))
ids, y, w = np.concatenate(ids), np.concatenate(y), np.concatenate(w)
X = np.eye(2)[ids]                                  # one-hot video id
ctr = LogisticRegression(C=np.inf, fit_intercept=False, tol=1e-10).fit(X, y)
wlr = LogisticRegression(C=np.inf, fit_intercept=False, tol=1e-10).fit(X, y, sample_weight=w)
for g, name in enumerate(videos):
    m = ids == g
    print(f"{name}: CTR {ctr.predict_proba(np.eye(2)[[g]])[0, 1]:.3f}, "
          f"odds {np.exp(wlr.coef_[0, g]):.3f}, sum(T)/(N-k) {w[m & y].sum() / (~y[m]).sum():.3f}")
# A (long video): CTR 0.039, odds 0.331, sum(T)/(N-k) 0.331
# B (clickbait): CTR 0.100, odds 0.220, sum(T)/(N-k) 0.220
# A click model ranks B first; the watch-time model ranks A first (E[T]: 0.32 vs 0.20).

# (d) A/B sample size per arm for watch time (sigma = 60 min, delta = 0.2 min)
z2 = (norm.ppf(0.975) + norm.ppf(0.8)) ** 2
print(f"n per arm {2 * z2 * 60**2 / 0.2**2:,.0f}")                # 1,412,798

# (e) PSI noise floor: no drift, 10 equal bins, 200 vs 200 samples
k, n_ref, m_live = 10, 200, 200
a = rng.multinomial(n_ref, np.full(k, 1 / k), size=20_000) / n_ref
b = rng.multinomial(m_live, np.full(k, 1 / k), size=20_000) / m_live
a, b = np.clip(a, 1e-4, None), np.clip(b, 1e-4, None)
psi = np.sum((b - a) * np.log(b / a), axis=1)
print(f"mean PSI {psi.mean():.3f}, formula {(k - 1) * (1 / n_ref + 1 / m_live):.3f}, "
      f"share above 0.1: {(psi > 0.1).mean():.2f}")
# mean PSI 0.092, formula 0.090, share above 0.1: 0.37

The notebook adds the in-batch softmax with and without the log-Q correction, and asks you to write each piece yourself.

6. Where it shows up #

• Interviews. Recommendations, feeds, search, ads, notifications and "people you may know" are all funnels with the template of section 4. Harmful-content and fraud detection use the same framework with a different funnel: cheap rules, then models, then human review.
• Industry papers. The YouTube funnel and weighted logistic regression (Covington et al., 2016), the two-tower model with the log-Q correction (Yi et al., 2019), and NE, calibration and downsampling for ads (He et al., 2014) are the standard references behind this lesson.
• ML for science. The funnel is the shape of many science pipelines. Virtual screening in drug discovery runs a fast filter or docking on millions of molecules and expensive simulations on a few hundred. Literature and dataset search for researchers is retrieval and ranking exactly as here. Analogue methods in weather and climate retrieve similar past atmospheric states, today often with learned embeddings and ANN search. Earthquake detection runs a cheap trigger (such as STA/LTA, the ratio of a short-term to a long-term average of the signal amplitude) before a slower phase picker. Drift monitoring applies when an instrument or an upstream model version changes.

7. Common mistakes #

• Starting with the model. The first 8 minutes are questions, metrics and numbers.
• No numbers. Without QPS, catalog size and a latency limit, no design choice can be defended. Ask for them, or state them as assumptions.
• Click as the only label. It rewards clickbait. Add watch time, likes and dismissals.
• Wrong negatives. Ranking negatives are impressions without engagement; random or in-batch items are for retrieval, and in-batch negatives need the log-Q correction.
• A random train/test split. It leaks the future; split by time.
• Uncalibrated probabilities where calibration matters. In ads the probability sets the price; in a multi-task score, heads trained on downsampled data must be corrected before they are combined.
• Offline gain = success. Only an A/B test with enough power and with guardrails decides.
• Randomizing by request when the metric is per user. Then each user sees both systems, and the per-user metric mixes them.
• Budgeting only the average latency. Parallel calls and heavy tails make p99 the real limit.
• A fixed PSI threshold on small windows. The noise floor $(k-1)(1/n + 1/m)$ can be close to the threshold; compute it before you choose the window.
• No plan for cold start, position bias, deployment or monitoring. These are the usual deep dives.

8. Check yourself #

1. 20 M daily active users, 8 requests per user per day, peak factor 3. Average and peak QPS?
   Answer

   $1.6 \times 10^8 / 86{,}400 \approx 1{,}852$ QPS on average; $\approx 5{,}556$ QPS at peak.
2. 200 M items with 64-dimensional float16 embeddings. Index memory? And with 16-byte PQ codes?
   Answer

   $2 \times 10^8 \times 64 \times 2$ B $= 25.6$ GB; with PQ, $2 \times 10^8 \times 16$ B $= 3.2$ GB.
3. An ads model trained with negatives kept at $w = 0.05$ predicts $p_s = 0.2$. What is the calibrated click probability?
   Answer

   $p = 0.2 / (0.2 + 0.8/0.05) = 0.2 / 16.2 \approx 0.0123$, about 1.2%.
4. Video A has a click rate of 4% and 8 minutes of mean watch time after a click; video B has 10% and 2 minutes. Which one does a click model rank first, and which one does the weighted logistic regression of Result 2 rank first?
   Answer

   The click model ranks B first ($0.10 > 0.04$). The weighted model learns odds $\approx \mathbb{E}[T] = P \times$ mean watch: $0.04 \times 8 = 0.32$ for A and $0.10 \times 2 = 0.20$ for B, so it ranks A first.
5. $k = 10$ bins, a reference of 10,000 rows, live windows of 500 requests. What is the PSI noise floor? And with windows of 100? What does this mean for a 0.1 threshold?
   Answer

   $9(1/10{,}000 + 1/500) \approx 0.019$; with $m = 100$, $9(0.0001 + 0.01) \approx 0.091$. A 0.1 threshold is safe for windows of 500 but gives many false alerts for windows of 100: use larger windows, fewer bins, or a higher threshold with hysteresis.
6. What is the difference between a shadow deployment and a canary release?
   Answer

   In a shadow deployment the new model gets a copy of real traffic, but users never see its answers: you check predictions, latency and errors with no user impact. In a canary release a small share of real users gets the new model's answers, so you can measure its effect on guardrails before a wider rollout, with an automatic stop and a rollback.

9. Interview angle #

Recognize the prompt.

The routine to say aloud. "Let me first clarify the users, the scale and the latency limit. Then I define the metrics at four levels. Then I size the system with numbers, draw the funnel, and go through data, models, serving and evaluation. I leave the last ten minutes for monitoring, deep dives and trade-offs."

Typical questions and short model answers.

1. "Why two stages?" — "Scoring 50 M items for each of about 11,600 requests per second with a $2 \times 10^7$-FLOP ranker needs about $10^{19}$ FLOP/s, hundreds of thousands of GPUs. So a cheap retrieval stage with precomputed item vectors and an ANN index picks about 500 candidates, and only these go to the ranker. Retrieval is judged by recall@500, the ranker by NDCG and per-user AUC."
2. "How do you train the retrieval model?" — "A two-tower model with a dot-product score, so item vectors are precomputed and served from an HNSW or IVF-PQ index. I train it with an in-batch softmax: the other items of the batch are negatives. Popular items appear as negatives too often, so I subtract $\log q_j$ from each logit, where $q_j$ is the item's frequency in the stream. Cross features stay in the ranker."
3. "Which label? Why not clicks?" — "Clicks reward clickbait. I train a multi-task ranker with click, watch time, like and dismiss heads, and combine them in a score whose weights I tune with A/B tests. For watch time, logistic regression with watch-time weights learns odds close to the expected watch time per impression."
4. "Offline AUC went up, online watch time did not. Why?" — "The label may not be the objective; the logged data come from the old system, with position bias and a feedback loop; there may be training–serving skew; the ranker only reorders the same candidates; or the test had too little power. I would check each one, starting with skew and the label."
5. "How do you ship the new model safely?" — "A written promotion gate on a frozen test set, including slices and latency. Then a shadow run on copied traffic, a canary at 1% and 5% with automatic stops on guardrails, and the A/B test with user randomization, a fixed sample size and at least two full weeks. Rollback is one alias move in the registry."
6. "How do you detect drift without labels?" — "PSI and KS on the key inputs and on the predictions. PSI has a noise floor of $(k-1)(1/n + 1/m)$, so I choose windows where it is far below the threshold, and alert with hysteresis. When labels arrive, I add calibration and NE on recent days. First I always rule out a pipeline bug: units, a new upstream source, a schema change."

Typical follow-ups. "How does a new video get recommended?" (content features in the item tower, a fresh pool, exploration slots). "What changes for ads?" (calibration, downsampling correction, delayed conversions, budget pacing). "Your p99 is too high." (timeouts per source, hedged requests, fewer candidates, a smaller ranker, caching). "How would you reduce the training cost?" (it is data loading and embedding lookups, not FLOPs: fewer negatives, feature caching, incremental daily training).

Mock prompt (search). "Design product search for an online shop: 10 M products, 2 M searches per day, p99 ≤ 300 ms." A strong answer contains: the numbers (about 23 QPS on average, about 70 with an assumed 3× peak; a 256-dimensional float32 index of 10.2 GB); query understanding (spelling, category intent); hybrid retrieval with BM25 and a two-tower semantic model, merged to about 1,000 candidates; a ranker on query–product features (BM25 score, embedding similarity, past click and add-to-cart rates for this query–product pair, price, stock); because the traffic is low, a small cross-encoder (a transformer that reads query and product together) on the top 100 (about $2 \times 2 \times 10^7 \times 128 \approx 5 \times 10^9$ FLOPs per pair for $2 \times 10^7$ parameters and 128 tokens, so about $3.6 \times 10^{13}$ FLOP/s at peak: one or two GPUs); labels from purchases, add-to-cart and clicks, plus human relevance grades for the test set; NDCG@10 offline; online search conversion, revenue per search and zero-result rate; position bias; new products through text and image embeddings; an A/B test by user.

10. Practice #

The notebook for this lesson runs in Colab and in the browser. Each exercise has a check cell and a folded solution:

1. ndcg_at_k: NDCG@k with both gains, checked against sklearn.metrics.ndcg_score.
2. back_of_envelope: QPS, index memory, ranker GPUs and log volume for any prompt.
3. in_batch_softmax_loss: the loss with and without the log-Q correction; then train a score table on a stream with a popularity skew and see which one recovers the true preferences.
4. psi and its noise floor: PSI from two samples with reference-quantile bins, and a simulation of the no-drift distribution.
5. ab_duration: days needed for an A/B test from the traffic, the metric's $\sigma$ and the minimum detectable effect, rounded up to full weeks.

Timed mock designs (45 minutes each, aloud, on paper; then check against the table of section 3.1: a complete design by minute 27, and a label, a metric and a number for every stage):

• the worked example of section 4, from memory;
• "Design ad click prediction for a social-media feed" (calibration, NE, downsampling, delayed conversions);
• the product-search mock prompt of section 9;
• "Design a notification system that decides which push notifications to send" (a funnel with a daily budget per user, and a guardrail on unsubscribes);
• "Design harmful-content detection for a video platform" (rules, models, human review; precision at a threshold).

Numbers drill. For each prompt above, compute QPS, index memory, ranker load and log volume in under 5 minutes, with rough arithmetic only.

Say it aloud drill. Answer each typical question of section 9 in 90 seconds, with a timer. Each answer needs one number. Record yourself and compare with the model answers.

11. Watch and read #

• Covington, Adams and Sargin (2016), Deep Neural Networks for YouTube Recommendations: §2 (the funnel), §3.4 (next-watch labels), §4.2 (weighted logistic regression).
• Yi et al. (2019), Sampling-Bias-Corrected Neural Modeling for Large Corpus Item Recommendations: §3 (in-batch softmax, the log-Q correction, streaming frequency estimation).
• He et al. (2014), Practical Lessons from Predicting Clicks on Ads at Facebook: §2 (NE and calibration), §3.2 (data freshness), §6 (negative downsampling and recalibration).
• Joachims, Swaminathan and Schnabel (2017), Unbiased Learning-to-Rank with Biased Feedback: the propensity-weighted estimator of section 3.7.
• Hello Interview, ML System Design in a Hurry: the delivery framework (free web pages): the time plan of section 3.1 and worked designs.
• Stanford CS 329S: Machine Learning Systems Design (Chip Huyen; free lecture notes and slides). Her book Designing Machine Learning Systems (O'Reilly, 2022) covers the same ground in depth.
• Goku Mohandas, Made With ML (free): versioning, testing, serving and monitoring, with code; the project of the ::: deeper block in section 3.8.
• Xu and Aminian, Machine Learning System Design Interview (ByteByteGo, 2023; a book, not free): chapter 1 (a seven-step framework) and the chapters on video recommendation, ad click prediction and the news feed.