GenAI system design: RAG, assistants, agents and text-to-image

Before this lesson: lesson 14 (the 45-minute framework and two-stage retrieval and ranking, which RAG reuses) and lesson 11 (fine-tuning, LoRA, LLM judges). Lesson 13 explains the serving numbers used here (prefill and decode, the KV cache, Little's law); more depth is in ML systems lesson 12. The contrastive loss behind dense retrieval is in information theory lesson 8; LLM evaluation in production is in MLOps lesson 9. Time: about 100 minutes, plus the notebook and one timed mock design.

---

Why this matters #

GenAI design is now a round of its own in ML engineer and research engineer loops: "Design a chatbot over our documentation", "Design a coding assistant", "Design a text-to-image product". A strong answer starts from the user and the metric, not from the model. It picks the cheapest adaptation that works, makes evaluation concrete, and gives numbers. A weak answer lists tools ("a framework, a vector database, a big model") without saying why or how much.

The same designs appear in ML for science. A lab or a weather service needs an assistant over its manuals, code and papers that cites its sources. A conditional diffusion model that generates weather fields or seismic ground motion from physical conditions has the same parts as text-to-image: a condition encoder, a latent space, guidance, and the step count as the cost lever.

1. Predict first #

2. The picture #

The simplest RAG system has three steps: embed the question, find the $k$ most similar text chunks, and put them into the prompt with the instruction "answer from these sources". Everything else in this lesson makes one of these steps better, cheaper, safer or measurable.

Why retrieve instead of training the facts into the model? An LLM stores knowledge in its weights only at training time, in a lossy way, and it cannot say where a fact came from. Retrieval gives the model fresh text at question time. So the knowledge can change daily, every claim can cite a source, and access rights can be checked before the model sees any text. The model's job becomes reading and writing, which it does well.

Two paths.

• Offline (ingestion), whenever documents change: parse → clean → split into chunks → embed each chunk → store vectors, text and metadata in a vector index and in a keyword index.
• Online, per question: rewrite the question with the chat history → retrieve candidates by keywords and by vectors → fuse the two lists → rerank the best candidates with a stronger model → assemble the prompt (instructions, history, chunks with ids, question) → generate with citations → check the output → stream it to the user.

The online path is a chain of stages, and the user waits for all of them before the first word appears. The time to first token (TTFT) is the time from the question to the first output token; the time per output token (TPOT) sets how fast the answer streams after that.

Interactive figure: loading…

The same skeleton covers the other systems.

• A chat assistant is RAG plus conversation state, guardrails and feedback.
• An agent replaces the single retrieve-then-generate pass by a loop: the model chooses a tool, the system runs it, and the result goes back into the context.
• A text-to-image service replaces the LLM by a diffusion or flow model: text encoder → iterative denoiser in a latent space → decoder → safety check.

The framework of lesson 14, adapted.

Step	What to say for a GenAI system
Clarify	users and tasks; which knowledge; freshness; access rights; languages; latency target (TTFT); traffic; budget; hosted API or self-hosted model
Metrics	product (resolution rate, retention), quality (correctness, faithfulness), latency (TTFT, TPOT), cost per request, safety (harmful outputs, data leaks)
Data	the corpus (RAG), demonstrations or preference pairs (fine-tuning), image–text pairs (text-to-image), and an evaluation set
Model	the adaptation ladder (section 3.1); model size from quality, latency and cost
Training	usually small parts only: the retriever, the reranker, a LoRA for style; the full recipe for text-to-image only if asked
Serving	a latency budget per stage; GPUs from tokens or images per second; caching
Evaluation	per component (retrieval, generation) and end to end; a fixed "golden" question set; an LLM judge checked by humans; an online A/B test
Monitoring	feedback, refusals and escalations, cost, latency, drift of question topics, safety incidents
Trade-offs	quality vs latency vs cost; freshness; risk

3. Definitions and main results #

3.1 The adaptation ladder: prompt, RAG or fine-tune? #

Climb only as far as you need. Each step costs more and changes something different.

	Prompting	RAG	Fine-tuning (e.g. LoRA)
Changes	instructions and examples	the text the model reads	behaviour stored in the weights
Good for	format, simple tasks	facts that change, long-tail facts, citations, access control	style, output format, a narrow skill, shorter prompts
Update cost	edit a text	re-index the changed documents	a new training run
Weakness	context length	retrieval misses	does not add facts reliably; can forget old skills

The rule for interviews: RAG for knowledge, fine-tuning for behaviour. The two combine well. Fine-tuning the embedding model or the reranker on in-domain (query, relevant chunk) pairs is often the largest single retrieval gain, while the generator stays general.

3.2 Ingestion: chunking #

Retrieval returns chunks, not whole documents, so the split matters. A chunk that is too large mixes topics: its one embedding is vague, and it fills the context. A chunk that is too small lacks the context needed for an answer.

Fixed-size chunks with overlap. Cut a document of $T$ tokens into chunks of $c$ tokens that overlap by $o$ tokens. The stride is $s = c - o$, chunk $j$ (counting from 0) starts at token $js$, and the number of chunks is

The last chunk starts at $(n-1)s \ge T - c$, so it reaches the end of the document.

Result 1 (what overlap guarantees). Every span of at most $o + 1$ consecutive tokens lies completely inside at least one chunk.

Proof.

1. Take a span that starts at token $a$ and has $\ell \le o + 1$ tokens, so it ends before token $a + \ell$.
2. Let $j = \lfloor a/s \rfloor$, so $js \le a \le js + s - 1$. Chunk $j$ covers tokens $js$ to $js + c - 1$.
3. The span ends at token $a + \ell - 1 \le (js + s - 1) + o = js + c - 1$, because $s + o = c$. So chunk $j$ contains the whole span.
4. If $j > n - 1$, use the last chunk instead: it starts at $(n-1)s \le a$ and runs to the end of the document. $\blacksquare$

So the overlap is not a free parameter: it is the longest answer span (in tokens) that you never want to cut. The price is extra tokens to embed and store, about a factor $c/s$.

Interactive figure: loading…

Better than fixed windows: split on structure (headings, paragraphs, list items, functions and classes in code), and fall back to fixed windows inside long sections. Store metadata with each chunk (document id, title, date, version, access group) for filtering and citation. Typical sizes are 200–800 tokens.

3.3 Retrieval: lexical, dense and hybrid #

Lexical retrieval (BM25) scores a chunk $d$ for a query $q$ by the query terms $t$ it contains:

Here $\text{tf}(t, d)$ is a saturated term frequency: $f(t,d)$ is the count of $t$ in $d$, $|d|$ the length of $d$ and avgdl the mean chunk length. The inverse document frequency $\text{IDF}(t)$ is large for rare terms; a common form is $\ln\!\big(1 + \frac{N - n_t + 0.5}{n_t + 0.5}\big)$, with $N$ chunks of which $n_t$ contain $t$. The parameter $k_1 \approx 1.2$–$2$ limits the gain from repeating a term, and $b \approx 0.75$ corrects for length. BM25 is strong on exact strings: error codes, product names, function names, ids.

Dense retrieval (bi-encoder). An encoder maps the query and each chunk to vectors $e_q, e_d \in \mathbb{R}^D$ ($D$ is the embedding dimension, for example 768). The score is the cosine similarity. Queries and chunks are encoded separately, so all chunk vectors are computed offline. The encoder is trained with the InfoNCE loss on a batch of $N$ pairs $(q_i, d_i)$:

where the other chunks of the batch are the negatives and $\tau$ is a temperature (the same loss as the two-tower model of lesson 14). Dense retrieval finds paraphrases ("can't log in" ↔ "reset your password") that BM25 misses.

Hybrid retrieval with reciprocal rank fusion (RRF). Run both retrievers and merge the two ranked lists. The raw scores have different scales (BM25 scores of about 5–20, cosines of about 0.7–0.9), so RRF uses only the ranks (Cormack et al., 2009):

A document missing from a list adds nothing for that list. With $k = 60$, the scores of nearby ranks are close, so a chunk ranked well by both lists beats a chunk ranked first by one list and low by the other. A small $k$ gives more weight to the very top of each list.

Interactive figure: loading…

Approximate nearest neighbours (ANN). Exact search compares the query with all $N$ vectors: $N D$ multiply-adds and $N D \beta$ bytes read per query, with $\beta$ bytes per number. Graph indexes (HNSW) and cluster indexes (IVF, optionally with product quantization) visit only a few thousand vectors. For a few million vectors they find 95–99% of the exact top $k$ in about a millisecond. Memory: $N D \beta$ for the vectors, plus about $2M$ neighbour ids of 4 bytes per vector for HNSW with parameter $M$.

Reranking. A cross-encoder reads the query and one chunk together in one transformer, so its attention can match words across them. It is more accurate than the bi-encoder, but it needs one forward pass per (query, chunk) pair, so it reranks only the top 20–100 candidates. This is the two-stage idea of lesson 14: a cheap stage for recall, an expensive stage for precision.

3.4 Prompt assembly and generation #

• Context budget. System prompt + history + chunks + question + output must fit in the context window. Every input token costs prefill compute: about $2P$ FLOPs per token for a model with $P$ parameters, plus causal attention, about $2 L D T^2$ FLOPs in total for $T$ tokens, $L$ layers and width $D$. More chunks raise recall but add noise, cost and latency.
• Order. Models use information at the start and the end of a long context better than in the middle ("lost in the middle", Liu et al., 2023). Put the best chunks first and keep $k$ small (3–8).
• Grounding. Instruct the model to answer only from the sources, to cite chunk ids, and to say "I don't know" (or hand over to a human) when the sources do not answer. Accept only citations of chunk ids that were actually retrieved.
• Access control. Filter chunks by the user's access groups in the retrieval query, before any text reaches the model. An instruction such as "do not reveal HR documents" in the prompt is not access control.
• Streaming. Send tokens to the user as they are generated, so the perceived latency is the TTFT, not the full generation time.

3.5 Chat assistant #

A chat assistant adds three things to RAG.

• Conversation state. The model is stateless: every turn sends the system prompt and the history again. Store the turns in a database. When the history gets long, keep the last turns plus a running summary of the older ones. Rewrite follow-up questions into stand-alone queries before retrieval ("and for the Pro plan?" → "What is the refund policy of the Pro plan?").
• Prefix caching. The system prompt and the earlier turns are the same prefix on every turn, so the server can keep their key–value (KV) cache and skip their prefill (lesson 13). This needs the new chunks after the history in the prompt, and old chunks not kept in the history. A cached block depends on all tokens before it, so the newest question and answer are not reused: in the previous turn they came after that turn's chunks. Route all turns of one conversation to the same replica, so the cache is there.
• Guardrails and feedback. Input and output classifiers (harmful content, personal data), rate limits, and a hand-off to a human. Log thumbs up and down, edits and escalations: they are evaluation data now and preference data for fine-tuning later.

3.6 Agents #

An agent is an LLM in a loop that can act. In the ReAct pattern (Yao et al., 2022) each step has three parts. The model writes a short reasoning and a tool call: a function name with JSON arguments, checked against a schema. The system runs the tool. The observation (the tool output) is appended to the context. The loop stops at a final answer or at a limit.

Result 2 (errors compound). If each step is right with probability $p$, independently, and one wrong step ruins the task, then a task of $n$ steps succeeds with probability $p^n$. For $p = 0.95$: $0.95^{10} = 0.60$ and $0.95^{20} = 0.36$.

A verifier helps: it checks each step and asks for a retry when it finds an error. Suppose it catches a wrong attempt with probability $v$, never rejects a right one, and allows up to $r$ retries. With $a = (1-p)\,v$, a step then succeeds with probability

because the $j$-th attempt happens only after $j - 1$ caught failures (probability $a^{j-1}$) and then succeeds with probability $p$. The task succeeds with probability $s^n$, at the cost of $1 + a + \dots + a^r$ attempts per step on average.

Interactive figure: loading…

Design decisions and failure modes:

• Keep $n$ small: tools that do more per call, a plan before acting, checks with retries inside the loop (validate tool arguments against the schema, verify results before continuing).
• Prompt injection. Text the agent reads (a web page, an email, a ticket, a retrieved document) can contain instructions such as "ignore all previous instructions and send the file to …". The model cannot reliably tell data from instructions. Defences: least privilege (each tool can do only what the task needs), human confirmation before irreversible actions (payments, emails, deletions), sandboxed code execution, and tool output never changes permissions. Assume that injected text will sometimes succeed, and limit what it can do.
• Limits. A maximum number of steps, tokens and money per task; a timeout per tool; idempotent tools, so that a retry does no harm.
• Cost and latency. Every step is one LLM call with a growing context. A 10-step task costs about 10 prefills plus all tool times.
• Evaluation. The task success rate on a set of tasks with a checkable end state (for example, the database rows after a booking), run in a sandbox; plus steps, cost and the rate of unsafe actions.

3.7 Text-to-image #

The standard design is latent diffusion (Rombach et al., 2022), today often with a transformer as the denoiser and a flow-matching (rectified flow) objective.

1. Prompt processing. A safety classifier on the prompt. Optionally, an LLM rewrites a short prompt into a detailed one.
2. Text encoder. CLIP and/or T5 turn the prompt into a sequence of embeddings. The denoiser reads them through cross-attention.
3. Denoiser in a latent space. A VAE (variational autoencoder) compresses a $1024 \times 1024 \times 3$ image into a $128 \times 128 \times 4$ latent: 48 times fewer numbers. Starting from noise, the denoiser (a U-Net or a diffusion transformer) is applied $S$ times, once per sampler step (often 20–50).
4. Classifier-free guidance (CFG). At each step, evaluate the denoiser with the prompt ($\varepsilon_c$) and with an empty prompt ($\varepsilon_\emptyset$), and combine

with a guidance scale $w > 1$ (often 5–8). A larger $w$ follows the prompt more closely but gives less diversity. The model learns both outputs because the prompt is dropped in 10–20% of the training examples (Ho and Salimans, 2022). Cost: two evaluations per step, usually one pass with batch 2. 5. Decode and check. The VAE decoder maps the latent to pixels. Then an image safety classifier, a watermark or provenance metadata, storage and a content delivery network.

Why CFG sharpens. The predicted noise is the score up to a factor: $\varepsilon \approx -\sigma\, \nabla_x \log p_\sigma(x)$ at noise level $\sigma$. By Bayes' rule, $\nabla_x \log p_\sigma(x \mid c) = \nabla_x \log p_\sigma(x) + \nabla_x \log p_\sigma(c \mid x)$. The guided combination therefore uses the score $\nabla_x \log p_\sigma(x) + w\,\nabla_x \log p_\sigma(c \mid x)$: the score of $p(x)\,p(c \mid x)^w$. With $w > 1$, the implicit classifier $p(c \mid x)$ counts $w$ times, so samples move to where the condition is clearest and away from the other classes. This is a picture, not an exact sampler: the samples are close to, but not exactly, samples of $p(x)\,p(c \mid x)^w$.

Interactive figure: loading…

The main cost lever is $S$. The compute per image is about $S$ times the cost of one CFG pass. One pass already processes thousands of latent positions (a $128 \times 128$ latent has 16,384), so the GPU is busy even at batch 1. So batching gives much less extra throughput than in LLM decoding, where one step processes a single token per sequence. To cut the cost: distilled few-step models (consistency or adversarial distillation, 1–4 steps), guidance distillation (one pass per step instead of two), lower precision, and a smaller image plus an upscaler.

Training data (if asked). Billions of image–text pairs, filtered by CLIP similarity, aesthetic score, safety and duplicates. Replacing short alt-text captions by detailed synthetic captions from a captioning model improves prompt following (as reported for DALL·E 3).

3.8 Evaluation #

Evaluate the components separately, then end to end. Otherwise you cannot say why an answer is wrong.

Retrieval (questions with labelled relevant chunks). For one question with relevant set $R$ and a ranked list:

• recall@$k$ = (relevant chunks in the top $k$) / $|R|$: did we find them?
• reciprocal rank = 1 / (rank of the first relevant chunk), and 0 if no relevant chunk is in the list; MRR is its mean over questions;
• nDCG@$k$ = DCG@$k$ / (DCG@$k$ of the ideal ranking), with binary labels: $\text{DCG@}k = \sum_{i=1}^{k} \text{rel}_i / \log_2(i+1)$, $\text{rel}_i \in \{0, 1\}$. It rewards relevant chunks near the top.

Generation.

• Correctness against a reference answer.
• Faithfulness (groundedness): every claim is supported by the retrieved chunks. A faithful answer can still be wrong, when the chunk is wrong or old. An unfaithful answer is a hallucination even when it happens to be right.
• Citation precision (each cited chunk supports its claim) and correct refusal on questions that the corpus cannot answer.

LLM as a judge. A strong model grades answers with a rubric. It scales, but it has biases: position (it prefers the first answer of a pair), length (it prefers longer answers) and self-preference (it prefers text from its own model family). Check it before trusting it: label 100–200 answers by humans, measure the agreement, swap the order in pairwise comparisons, and check again when the judge model changes. Cohen's kappa measures agreement beyond chance:

where $p_o$ is the observed share of answers on which judge and human agree, and $p_e$ is the share expected by chance from the two label frequencies ($p_e = \sum_\ell p^{\text{judge}}_\ell\, p^{\text{human}}_\ell$). $\kappa = 1$ is perfect agreement, $\kappa = 0$ is chance level. Example: on 100 answers both say "pass" 70 times and "fail" 15 times; the judge says pass 80 times, the human 75 times. Then $p_o = 0.85$, $p_e = 0.8 \times 0.75 + 0.2 \times 0.25 = 0.65$, and $\kappa = 0.20/0.35 = 0.57$: moderate agreement, although the raw agreement of 85% looks high.

Result 3 (is system B better than A? A paired test). Run both systems on the same $n$ questions. Let $b$ be the number of questions that A fails and B passes, and $c$ the number that A passes and B fails. Under $H_0$ (no difference), $c \sim \text{Binomial}(b + c, 1/2)$, and the two-sided p-value is $\min\{1,\ 2\,P(X \le \min(b, c))\}$ for $X \sim \text{Binomial}(b + c, 1/2)$: McNemar's exact test.

Why.

1. Questions that both systems pass, or both fail, say nothing about the difference.
2. Under $H_0$, each of the $b + c$ discordant questions is equally likely to favour A or B, independently of the others.
3. So the number $c$ that favour A is $\text{Binomial}(b + c, 1/2)$, and the p-value is the two-sided tail of this distribution. $\blacksquare$

An unpaired test on the two pass rates treats the variation in question difficulty as noise and loses power (section 4 shows both).

Images. FID (Fréchet inception distance) fits Gaussians to Inception-network features of real and generated images and computes $\|\mu_1 - \mu_2\|^2 + \operatorname{tr}\big(\Sigma_1 + \Sigma_2 - 2(\Sigma_1\Sigma_2)^{1/2}\big)$: realism and diversity; it needs thousands of images. CLIP score is the cosine similarity of the CLIP image and text embeddings: prompt alignment. Add compositional tests (counts, colours, positions), and a human side-by-side preference study on a fixed prompt set, which is still the final judge. Plus the rate of unsafe outputs on red-team prompts.

Online. An A/B test on the product metric (resolution rate, images downloaded) with guardrail metrics (complaints, escalations, latency, cost).

Answers to section 1. (1) RAG, not fine-tuning, for the knowledge. Fine-tuning would need a new run every day, cannot cite sources, cannot enforce access rights (every user would get everything the weights store), and does not store facts reliably. Index the documents with their access groups, filter at retrieval time, and cite chunk ids. Fine-tune only where behaviour must change: the retriever or reranker on internal query–document pairs, or the answer style. (2) $0.95^{10} = 0.599$: only about 60% of the tasks succeed, although each step is 95% reliable.

4. Worked example: a support assistant, then text-to-image sizing #

Prompt (45 minutes). "Design a customer-support assistant for a software company. Knowledge: 20,000 help-center articles (about 1,500 tokens each) and 2 million resolved support tickets (about 300 tokens each). Traffic: 50,000 conversations per day, 5 user turns each; the peak hour has 4× the average traffic. Answers must link to their sources. p95 TTFT ≤ 1.5 s. Self-hosted open-weights models on H100 GPUs." All numbers are an invented but realistic example.

(a) Clarify and metrics. Users are customers; the scope is how-to and troubleshooting questions. Account changes are out of scope for version 1 (they would make it an agent; see (g)). Product metric: the share of conversations resolved without a human, and customer satisfaction (CSAT). Guardrails: complaints about wrong answers, and escalations that came too late.

(b) Data and index.

• Articles: $c = 512$, $o = 64$, stride 448, so $\lceil (1500 - 512)/448 \rceil + 1 = 3 + 1 = 4$ chunks per article (starting at tokens 0, 448, 896, 1344): 80,000 chunks.
• Tickets are noisy and contain personal data. Offline, an LLM turns each resolved ticket into a short question–answer pair (about 200 tokens) and removes the personal data: 2 million chunks.
• Total $N = 2.08$ million chunks. With $D = 768$ float32 embeddings: $2.08 \cdot 10^6 \times 768 \times 4\ \text{B} = 6.4$ GB (3.2 GB in fp16), plus $2.08 \cdot 10^6 \times 32 \times 4\ \text{B} = 0.27$ GB of HNSW links ($M = 16$). This fits in the memory of one server; two replicas give availability. No distributed vector database is needed.
• Exact search would read 6.4 GB per query: about 0.3 s at 20 GB/s of memory bandwidth. HNSW takes about a millisecond.
• Embedding the corpus: about $3.4 \cdot 10^7$ article tokens (with overlap) plus $4 \cdot 10^8$ ticket tokens, $4.3 \cdot 10^8$ in total, through a 110 M-parameter encoder: $2 \times 1.1 \cdot 10^8 \times 4.3 \cdot 10^8 \approx 9.5 \cdot 10^{16}$ FLOPs, about 16 minutes on one GPU at 100 TFLOP/s achieved. Re-embedding is cheap: changed documents every night, the whole corpus when the encoder changes.

(c) Online path and latency budget. Question rewrite with a small model (assume 150 ms) → BM25 top 50 (about 20 ms) and dense top 50 (about 30 ms) in parallel → RRF → cross-encoder on 50 pairs (assume 60 ms on a GPU) → the best 5 chunks → prompt → LLM. Prompt per turn: system 400 + history 800

• 5 chunks × 500 + question 50 = 3,750 tokens; output about 200 tokens. LLM: a 14.8 B-parameter model with grouped-query attention (48 layers, 8 KV heads of dimension 128; the shape of Qwen2.5-14B), 29.5 GB of bf16 weights, one H100 per replica. Prefill: $2 \times 1.48 \cdot 10^{10} \times 3750 = 1.1 \cdot 10^{14}$ FLOPs, which takes 0.28 s at 40% of 989 TFLOP/s. (We ignore the attention term $2 L D T^2 \approx 7 \cdot 10^{12}$ FLOPs, with $L = 48$ layers and width $D = 5120$: 6% at this length. It grows with $T^2$ and reaches 27% at 16,000 tokens.) TTFT $\approx 0.15 + 0.03 + 0.06 + 0.28 \approx 0.52$ s plus queueing: inside 1.5 s (the first figure shows this timeline). We also set a TPOT target of 30 ms, so the answer streams fluently.

(d) Capacity at peak. $250{,}000$ turns per day: 2.9 turns/s on average, 11.6 turns/s at peak.

• Prefill: $11.6 \times 1.1 \cdot 10^{14} = 1.3 \cdot 10^{15}$ FLOP/s, that is 3.2 H100s at 40% of peak. With prefix caching of the system prompt and the earlier turns ($400 + 800 - 250 = 950$ of the 3,750 tokens; the newest question and answer, about 250 tokens, are not cached): 2.4 GPUs.
• KV memory: $2 \times 48 \times 8 \times 128 \times 2$ B = 192 KiB per token (keys and values, layers, KV heads, head dimension, bytes). By Little's law (requests in the system = arrival rate × time in the system): a turn lasts at most about $0.28 + 200 \times 0.03 \approx 6.3$ s, so at most about $11.6 \times 6.3 \approx 73$ turns are active at peak. At about 3,850 tokens each, they hold 55 GB of KV cache: two GPUs by memory (about 45 GB free on each after the weights and 5 GB for activations).
• Decode: $11.6 \times 200 = 2{,}300$ output tokens/s. Decode is limited by memory bandwidth: each step reads all weights and the KV cache of the batch. With 36 sequences per GPU, a step reads $29.5 + 36 \times 3{,}850 \times 192\ \text{KiB} \approx 56.7$ GB, which takes $56.7/3{,}350 \approx 17$ ms at 3.35 TB/s (below the 30 ms target). That is about 2,100 tokens/s per GPU: about one GPU of decode work.
• Total: about $2.4 + 1.1 = 3.5$ GPUs busy at peak, so 5 H100s at a 70% utilization target and a sixth as a spare. Prefill, not decode, is the main load: RAG prompts are long and answers are short.

(e) Evaluation plan.

• Golden set: 500 real questions sampled from tickets per product area, with expert reference answers and labelled relevant chunks; 10% are questions that the corpus cannot answer.
• Retrieval: recall@5, MRR and nDCG@10. For one question with relevant chunks $\{c_2, c_7, c_9\}$ and the ranking $c_4, c_2, c_8, c_7, c_1$: recall@5 = 2/3, reciprocal rank = 1/2, and nDCG@5 $= \dfrac{1/\log_2 3 + 1/\log_2 5}{1 + 1/\log_2 3 + 1/\log_2 4} = \dfrac{1.062}{2.131} = 0.498$.
• Answers: correctness, faithfulness, citation precision and correct refusal, graded by an LLM judge with a rubric whose agreement with humans was checked on 200 labelled answers.
• Comparison: a new reranker raises the pass rate on 300 questions from 70% to 76%. 30 questions went from fail to pass ($b = 30$) and 12 from pass to fail ($c = 12$). McNemar's exact test: $2\,P(X \le 12)$ for $X \sim \text{Binomial}(42, 1/2)$ gives $p = 0.008$, so the gain is significant. An unpaired two-proportion test on the same pass rates gives $z = 1.66$, $p = 0.10$: pairing removes the variation in question difficulty.
• Online: an A/B test on the resolution rate and CSAT, with escalation and complaint rates as guardrails. Monitor the no-answer rate, thumbs-down, question topics after each product release, latency and GPU cost.

(f) Failure modes and fixes.

• Retrieval misses and the model guesses → grounding instruction, refusal plus hand-off; track the no-answer rate.
• Old article versions → version and date metadata; filter by the customer's product version.
• Prompt injection inside tickets (customers wrote them) → treat chunks as data, no tools in version 1, an output filter.
• Personal data from tickets → remove it at ingestion, not at answer time.
• Long conversations → a summary of the old turns.

(g) Follow-up: "Let it also cancel subscriptions." Now it is an agent. Add a tool cancel_subscription(account_id) that can touch only the logged-in user's account, ask the user for explicit confirmation before the call, make the API idempotent, and write an audit log. Evaluate on sandboxed tasks with a checkable end state.

4b. Text-to-image sizing #

Prompt. "A design app generates 1024 × 1024 images: 2 million images per day, peak 3× the average, p95 latency ≤ 10 s for a request of 4 images."

• Traffic: $2 \cdot 10^6 / 86{,}400 = 23$ images/s on average, 69 images/s at peak.
• Model: an SDXL-class latent diffusion model (2.6 B-parameter U-Net), 30 steps with CFG: about 2.5 s per image on one A100 with bf16, fused attention and a compiled model (a PyTorch blog post measured 2.5 s for this setting).
• GPUs: $69.4 \times 2.5 = 174$ GPUs busy at peak, so 248 A100s at a 70% utilization target.
• Latency: 4 images on one GPU take about 10 s, because a compute-bound denoiser gains little from batching. Spread the 4 images over 4 GPUs: about 2.5 s.
• Lever: a 4-step distilled model. Estimate its time as 4/30 of the time: about 0.33 s per image, so about 23 GPUs busy and 33 at 70%, about 7 times fewer. This estimate ignores two effects in opposite directions: the text encoders and the VAE decoder keep their fixed cost, and many distilled models need no CFG pass (half the denoiser cost per step). Before switching, compare FID, CLIP score and human preference.
• Storage: about 1.5 MB per PNG, so 3 TB per day. Keep the images that users save; expire the rest.
• Evaluation: FID and CLIP score on a fixed prompt set as offline checks; a human side-by-side study on about 1,000 prompts before each model change; the unsafe-output rate on red-team prompts. Online: the share of requests with at least one downloaded image, and the regenerate rate.

5. In code #

The numbers of section 4, checked with plain Python, scikit-learn and SciPy:

import math
from scipy.stats import binomtest, norm
from sklearn.metrics import ndcg_score

# (a) Chunks and index size of the worked example
def n_chunks(T, c, o):                    # a document of T tokens, chunks of c tokens, overlap o
    return 1 if T <= c else math.ceil((T - c) / (c - o)) + 1

N = 20_000 * n_chunks(1500, 512, 64) + 2_000_000      # article chunks + ticket Q&A pairs
print(n_chunks(1500, 512, 64), N)                     # 4 2080000
print(round(N * 768 * 4 / 1e9, 2), round(N * 2 * 16 * 4 / 1e9, 2))   # 6.39 0.27 (GB of vectors, of HNSW links)

# (b) Retrieval metrics for one question
ranked = ["c4", "c2", "c8", "c7", "c1"]               # top 5 returned, best first
relevant = {"c2", "c7", "c9"}
hits = [d in relevant for d in ranked]
print(round(sum(hits) / len(relevant), 3), 1 / (hits.index(True) + 1))   # 0.667 0.5 (recall@5, RR)
docs = [f"c{i}" for i in range(1, 11)]                # the whole (toy) corpus
y_true = [[int(d in relevant) for d in docs]]
y_score = [[5 - ranked.index(d) if d in ranked else 0 for d in docs]]
print(round(ndcg_score(y_true, y_score, k=5), 3))     # 0.498

# (c) Reciprocal rank fusion: chunk A has ranks (1, 10), chunk B has ranks (3, 3)
def rrf(ranks, k=60):
    return sum(1 / (k + r) for r in ranks)
print(round(rrf([1, 10]), 5), round(rrf([3, 3]), 5))  # 0.03068 0.03175: B comes first

# (d) Is B better than A? The same 300 questions: 70% -> 76% pass
b, c = 30, 12                                         # fail->pass, pass->fail
print(round(binomtest(c, b + c, 0.5).pvalue, 4))      # 0.0079: paired (McNemar's exact test)
pool = (0.70 + 0.76) / 2
z = 0.06 / math.sqrt(pool * (1 - pool) * 2 / 300)
print(round(z, 2), round(2 * norm.sf(z), 3))          # 1.66 0.098: unpaired test, loses power

# (e) Agents: all n steps right with probability p**n
print(round(0.95**10, 3), round(0.95**20, 3))         # 0.599 0.358

# (f) LLM capacity of the worked example (14.8 B model, H100 at 40% of 989 TFLOP/s)
P, T_in, peak_turns = 14.77e9, 3750, 4 * 250_000 / 86_400
print(round(2 * P * T_in / (0.4 * 989e12), 2))                       # 0.28 s of prefill per turn
print(round(peak_turns * 2 * P * (T_in - 950) / (0.4 * 989e12), 1))  # 2.4 GPUs of prefill, prefix cache
kv_per_token = 2 * 48 * 8 * 128 * 2                                  # K and V, layers, KV heads, head dim, bytes
print(kv_per_token / 1024, round((29.5e9 + 36 * 3850 * kv_per_token) / 3.35e12 * 1e3, 1))
# 192.0 KiB per token; 16.9 ms per decode step with 36 sequences

# (g) Text-to-image GPUs at peak (2.5 s per image at 30 steps, 70% utilization target)
peak = 3 * 2e6 / 86_400                               # images per second
print(round(peak * 2.5 / 0.7), round(peak * 2.5 * 4 / 30 / 0.7))    # 248 33 (30 steps, 4 steps)

In a real system you compute the retrieval metrics over the whole golden set and report their means with a confidence interval, per product area. The notebook asks you to write BM25, RRF, the retrieval metrics, Cohen's $\kappa$ and McNemar's test yourself.

6. Where it shows up #

• Industry. Support assistants, internal search, coding assistants and workflow agents are the most common GenAI products. The interview prompt is usually one of these, or "design ChatGPT" and "design an image generator".
• Papers behind the designs. RAG (Lewis et al., 2020), ReAct (Yao et al., 2022), latent diffusion (Rombach et al., 2022), classifier-free guidance (Ho and Salimans, 2022), reciprocal rank fusion (Cormack et al., 2009) and "lost in the middle" (Liu et al., 2023).
• ML for science. An assistant over a lab's or a weather service's manuals, code and papers; literature search that cites its sources; agents that run simulation or analysis tools in a sandbox, with limits.
• Conditional generative models for science. A latent diffusion or rectified-flow model of weather fields or seismic ground motion has the same parts as text-to-image: an encoder for the condition (the current atmospheric state; magnitude, distance and site), a latent space, guidance, and the step count as the cost lever. Evaluation asks the same questions with scientific scores: distribution distances for realism, fidelity to the condition, and the spread and calibration of the ensemble.

7. Common mistakes #

• Starting with the model before the users, the metric and the constraints are clear.
• Fine-tuning to add knowledge that changes, needs citations or has access rights.
• Dense retrieval only. It misses exact identifiers; add BM25 and fuse.
• Adding raw BM25 and cosine scores. Their scales differ; fuse by ranks (RRF) or calibrate first.
• Filling the context with 20+ chunks: noise, cost, latency, and "lost in the middle".
• Access control in the prompt instead of filtering before retrieval results reach the model.
• Only end-to-end evaluation, or an LLM judge never checked against humans, or an unpaired test for two systems run on the same questions.
• Agents with broad permissions, no confirmation before irreversible actions, and no limit on steps or cost.
• Text-to-image: forgetting that CFG doubles the denoiser cost, or expecting batching to make diffusion cheap, as it does for LLM decoding.

8. Check yourself #

1. 100,000 documents of 2,000 tokens each are cut into chunks of 400 tokens with 50 tokens of overlap and embedded with 1,024-dimensional fp16 vectors. How many chunks, and how much vector memory?
   Answer

   Stride 350, so $\lceil (2000 - 400)/350 \rceil + 1 = \lceil 4.57 \rceil + 1 = 6$ chunks per document and 600,000 chunks. Memory: $600{,}000 \times 1024 \times 2$ B $= 1.23$ GB, plus the index structure.
2. Chunk A is ranked 1st by BM25 and 10th by dense retrieval; chunk B is 3rd in both. Which one does RRF ($k = 60$) put first? Why does RRF use ranks and not scores?
   Answer

   $\text{RRF}(A) = 1/61 + 1/70 = 0.03068$, $\text{RRF}(B) = 2/63 = 0.03175$, so B comes first: agreement between the retrievers wins. BM25 scores and cosine similarities have different, query-dependent scales, so adding them needs a calibration; ranks do not.
3. With chunks of 300 tokens and an overlap of 40 tokens, what is the longest answer span that is always inside one chunk? What does doubling the overlap cost?
   Answer

   $o + 1 = 41$ tokens (Result 1). With $o = 80$ the stride falls from 260 to 220, so a long document needs about $260/220 = 1.18$ times as many chunks: about 18% more vectors to embed and store.
4. A text-to-image model samples 30 steps with classifier-free guidance. How many denoiser evaluations does one image need? Name two ways to reduce the cost.
   Answer

   60 evaluations (30 passes with batch 2: with and without the prompt). A distilled few-step model (consistency or adversarial distillation); guidance distillation (one evaluation per step); also lower precision, or a smaller resolution with an upscaler.
5. A judge and a human agree on 90 of 100 answers. Both say "pass" 85% of the time. Is the judge good?
   Answer

   Not necessarily. $p_e = 0.85^2 + 0.15^2 = 0.745$, so $\kappa = (0.90 - 0.745)/(1 - 0.745) = 0.61$: moderate agreement. Most of the 90% comes from both saying "pass" often. Look at the agreement on failures, which matter most.
6. Each step of an agent is right with probability 0.99, and a task takes 10 steps. What is the task success rate, and what are two ways to raise it?
   Answer

   $0.99^{10} = 0.904$. Fewer steps (tools that do more per call, a better plan), and checks with retries inside the loop (validate tool arguments against a schema, verify results before continuing).

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. When do you choose RAG, and when fine-tuning?

   Show answer

2. Why is RAG better than fine-tuning for a company's changing documents?

   Show answer

3. Write the reciprocal rank fusion score.

   Show answer

4. What does a cross-encoder reranker add to a bi-encoder retriever?

   Show answer

5. Chunks of size $c$ with overlap $o$: which answer spans are always inside one chunk?

   Show answer

6. Each agent step is right with probability $p$; what is the success rate of an $n$-step task? Compute it for $p = 0.95$, $n = 10$.

   Show answer

7. What is the main defence against prompt injection in an agent?

   Show answer

8. Write the classifier-free guidance formula.

   Show answer

9. What is the main cost lever of text-to-image serving?

   Show answer

10. How do you check that an LLM judge can be trusted?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

Recognize the prompt.

The prompt says	Start with
"chatbot over our documents", "internal search with answers"	RAG: ingestion, hybrid retrieval, rerank, grounded prompt with citations, access filters
"RAG or fine-tuning?"	the adaptation ladder: knowledge → RAG, behaviour → fine-tuning; often both
"design ChatGPT", "a chat assistant"	conversation state, context budget, prefix caching, guardrails, feedback; GPUs from prefill and decode
"an agent that books / buys / changes things"	ReAct loop, $p^n$, limits, least privilege, confirmation, sandboxed evaluation
"design Midjourney", "text-to-image"	latent diffusion, CFG, steps as the cost lever, safety filters, GPU sizing
"how do you evaluate it?"	component metrics, golden set, judge with $\kappa$, paired test, online A/B

A plan for 45 minutes.

1. 0–5 min: clarify users, tasks, knowledge, freshness, access rights, latency, traffic, budget.
2. 5–10 min: metrics (product, quality, latency, cost, safety) and the evaluation set.
3. 10–25 min: architecture: draw the offline and the online path; name each component and why.
4. 25–32 min: numbers: chunks and index size, tokens per request, prefill and decode GPUs (or images per second and GPU-seconds per image), latency budget.
5. 32–40 min: evaluation (per component, judge, paired comparison, online test) and monitoring.
6. 40–45 min: failure modes, trade-offs, follow-ups.

Typical questions and short model answers.

1. "RAG or fine-tuning for our documentation bot?" — "RAG for the knowledge: the docs change, answers need citations, and access rights must be enforced before the model sees text. Re-indexing is cheap; a training run is not, and fine-tuned weights do not store facts reliably. I would fine-tune only the retriever or reranker on our query–document pairs, or the generator for style, later."
2. "Retrieval quality is poor. What do you try first?" — "I look at failures by type on the golden set. Usually: add BM25 to dense retrieval and fuse with RRF, so exact ids are found; add a cross-encoder reranker on the top 50; fix chunking and metadata filters; then fine-tune the embedding model on in-domain pairs from click or ticket logs. I measure each change with recall@k and nDCG."
3. "How do you stop hallucinations?" — "You cannot stop them fully. Ground the answer in retrieved text, allow 'I don't know' and a hand-off, accept only citations of retrieved chunk ids, measure faithfulness on a golden set with a checked judge, and show sources to the user."
4. "The context window is 1 M tokens. Why not put all documents in the prompt?" — "Prefill cost and latency grow with every input token, about $2P$ FLOPs per token; accuracy drops for information in the middle; and access control and freshness still need a retrieval step. Retrieval of 5 good chunks is cheaper and usually better."
5. "How do you protect an agent from prompt injection?" — "Assume some injections will succeed. Least privilege per tool, scoped to the user and the task; human confirmation before irreversible actions; sandboxed code execution; tool output never changes permissions; limits on steps and money; and an audit log. Prompt rules and input filters are extra layers."
6. "How do you evaluate a text-to-image model?" — "FID for realism and diversity on thousands of images, CLIP score for prompt alignment, compositional tests for counts, colours and positions, and a human side-by-side study on a fixed prompt set as the final decision. Plus the unsafe-output rate on red-team prompts, and online the download and regenerate rates."

Typical follow-ups. "How many GPUs?" (prefill FLOPs at peak at 40% of peak; KV memory by Little's law; decode by memory bandwidth). "How do you make it cheaper?" (a smaller model with RAG, prefix caching, fewer chunks, a router that sends easy questions to a small model, quantization). "Is the new version better?" (paired test on the golden set, then an A/B test). "What if a document must be deleted?" (delete its chunks from both indexes; RAG makes deletion possible, fine-tuning does not).

10. Practice #

The notebook for this lesson runs in Colab and in the browser. Each exercise has a check cell and a folded solution:

1. bm25_scores: BM25 for a query over a toy corpus, checked against a hand-computed case.
2. reciprocal_rank_fusion: fuse several ranked lists, and reproduce the A-versus-B example.
3. recall_at_k and mrr: retrieval metrics over a small golden set.
4. cohen_kappa: judge–human agreement, checked against sklearn.metrics.cohen_kappa_score.
5. mcnemar_exact: the paired test of Result 3, checked against scipy.stats.binomtest; then compare it with a paired bootstrap.

Timed mock designs (45 minutes each, aloud, on paper; then check against the framework table of section 2 and steps (a)–(g) of section 4):

• "An assistant over a research institute's 300,000 papers and technical reports that answers with citations; 500 researchers; some reports are embargoed."
• "A text-to-image feature in a slide editor: 5 million images per day, 512 × 512 and 1024 × 1024, latency ≤ 4 s."
• "An agent that books meeting rooms and travel for employees."
• "Design ChatGPT for 10 million daily users" (state, routing, prefix caching, GPU count, safety).

Say it aloud drill. Answer each typical question of section 9 in 90 seconds, with a timer. Every answer needs one mechanism and one number. Record yourself and compare with the model answers.

11. Watch and read #

Stanford CS25: V3 I Retrieval Augmented Language ModelsStanford Online · 79 min

Douwe Kiela, a co-author of the RAG paper, on retrievers, rerankers and when to train which part.

[1hr Talk] Intro to Large Language ModelsAndrej Karpathy · 60 min

From 45:43: jailbreaks, prompt injection (51:30) and data poisoning, the security part of section 3.6.

How AI Image Generators Work (Stable Diffusion / Dall-E) - ComputerphileComputerphile · 17 min

The denoising loop, text conditioning and classifier-free guidance of section 3.7, without formulas.

How does Stable Diffusion work? – Latent Diffusion Models EXPLAINEDAI Coffee Break with Letitia · 13 min

The VAE latent space, cross-attention to the text encoder, and why diffusion in a latent space is cheaper.

What is reciprocal rank fusion in hybrid search?Abhishek Thakur · 8 min

RRF on two ranked lists, with the role of the constant k; compare with the figure of section 3.3.

• Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.
• Yao et al. (2022), ReAct: Synergizing Reasoning and Acting in Language Models.
• Liu et al. (2023), Lost in the Middle: How Language Models Use Long Contexts.
• Rombach et al. (2022), High-Resolution Image Synthesis with Latent Diffusion Models: §3 (the latent space and conditioning by cross-attention).
• Ho and Salimans (2022), Classifier-Free Diffusion Guidance.
• Cormack, Clarke and Büttcher (2009), Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods.
• Sentence-Transformers documentation, Retrieve & Re-Rank: bi-encoder retrieval plus cross-encoder reranking, with code.
• PyTorch blog, Accelerating Generative AI Part III: Diffusion, Fast: the SDXL timing of section 4b, step by step.
• Chip Huyen, AI Engineering (O'Reilly, 2025; a book, not free): the chapters on evaluation, RAG and agents, and fine-tuning versus RAG. Aminian and Sheng, Generative AI System Design Interview (ByteByteGo, 2024; a book, not free): worked designs of a chatbot, RAG and text-to-image.