RAG (Retrieval-Augmented Generation) is the most common AI topic in interviews today because it is the spine of almost every production AI product. The pattern: take a user question, retrieve relevant chunks from a knowledge base, and ask the LLM to answer using only those chunks. Sounds simple. The failure modes are not.
The strongest candidates don’t just explain the architecture. They can troubleshoot it, identify bottlenecks, isolate failures, and explain trade-offs.
What? A “hallucination” is any LLM answer that is factually wrong or unsupported by the retrieved context. It is a symptom, not a root cause.
Why? Because interviewers ask this to learn whether you debug the data layer first or jump to “switch to a better model.” Mature engineers start with retrieval, not generation.
How? The audit ladder, in order:
category=fees filter is applied, so the system pulls loan content.MiniLM-L6-v2, query uses mpnet-base-v2 — vectors are 384-d vs 768-d, comparison is meaningless.Scenario 1 — “Works for everyone except Customer #42” (metadata bug)
The user’s question was about their specific product. The retriever retrieved generic docs. Diagnosis: customer_id filter is missing from the SQL WHERE clause. Fix: add WHERE customer_id = ? to the retrieval query.
Scenario 2 — “Suddenly worse after yesterday’s deploy” (embedding drift)
The team swapped embedding models from MiniLM to OpenAI text-embedding-3-small but the document embeddings remained MiniLM. Diagnosis: comparing vectors from two different models. Fix: re-ingest all documents with the new embedding model in a single blue/green index swap.
Scenario 3 — “Works at 9 AM, terrible at 9 PM” (latency / cache eviction) RAG latency rose above the SLO, retrieval returned slow or no results. The chunks are correct but only 80 of the 100 retrieved make it to the LLM. Diagnosis: the prompt context window was hit; chunks were silently dropped right-to-left. Fix: route the dropped chunk metadata to a debug log; cap top-k to fit the budget; raise number of returned chunks only after evaluating the impact on precision.
What? Chunking is how you split a document into retrievable units. Each unit becomes one row in the vector DB.
Why? Because every chunking mistake compounds: one bad boundary produces one bad retrieval, which produces one bad answer, multiplied by every user query that hits it.
How? Four rules:
\n\n → \n → . → space), not just character-count.Page 4 without re-prompting.Scenario 1 — FAQ PDF with bullet points
Pick: RecursiveCharacterTextSplitter(separators=["\n\n", "\n", ". ", " "]). Bullets become one chunk each; each answer carries its own metadata block (“FAQ: account opening”).
Scenario 2 — 200-page regulatory document Pick: hierarchical chunking — section-level chunks (1000-3000 tokens) containing smaller paragraph chunks (300 tokens). The router picks section first, then drills down. Avoids losing context that fixed-size chunking destroys.
Scenario 3 — Streaming JSON / semi-structured data
Pick: JSON-aware splitter that respects object boundaries. A simple length-based splitter would cut a JSON object in half and produce garbage embedding. Use langchain.text_splitter.RecursiveJsonSplitter or equivalent.
What? Vector databases return “nearest neighbours” using a similarity metric. Three common ones: cosine, L2 (Euclidean), inner product.
Why? Because choosing the wrong metric silently degrades retrieval quality without any error message.
How? Rule of thumb:
<=>): default. Document lengths vary; you care about direction not magnitude.<#>): use when embeddings are pre-normalised and you want the highest dot-product equivalent to highest similarity.Scenario 1 — Mixed-length documents (FAQs, PDFs, articles) Pick: cosine. A long document and a short document embedded at the same model produce different magnitudes but the same direction. Cosine handles this cleanly.
Scenario 2 — Image embeddings, fixed length Pick: cosine. Image embeddings tend to live on a unit sphere; magnitude is meaningless.
Scenario 3 — Pre-normalised user embeddings (face recognition mirrors)
Pick: inner product. After L2-normalisation, cosine = inner product; using <#> is faster because no normalisation is computed.
What? A symptom where top_k returns zero or low-similarity chunks. Often mistaken for a model bug.
Why? Because the fix is almost never “change the LLM” — it is data plumbing. Interviewers test whether you walk the layers.
How? Walk three layers, top to bottom:
SELECT COUNT(*) FROM chunks WHERE embedding IS NOT NULL; — emptiness here means ingestion failed silently.Scenario 1 — Index never re-built after schema change
The chunks.content column was renamed to chunks.body, but the ingest.py script still writes to old column. New rows are silently dropped. Fix: idempotent ingestion with a source_hash column so re-ingestion is observable.
Scenario 2 — Filter too aggressive
Production code added category IN ('faq', 'faq_extended') to filter, but the orange of queries lives in category=help. Top-k returns zero. Fix: allowlist the missing category; rebuild the test corpus to catch this.
Scenario 3 — Multi-tenant data leak / leakage filter blocks legit hits
A noisy-neighbour tenant caused the team to add a tenant-isolation filter. The filter is over-eager and excludes the current customer’s chunks. Fix: the filter should be WHERE tenant_id IN (...) not WHERE NOT IN (...).
What? A measurement programme that tells you whether your retrieval is good enough. Without it, you ship guessing.
Why? Because you cannot improve what you cannot measure, and interviewers love engineers who set up measurement first.
How? Three pillars:
Scenario 1 — “Our LLM is great, retrieval is bad” Retrieval-only metrics show precision@5 = 0.4. You build a golden set of 200 question/chunk-code pairs, run it weekly. Iterating on chunking strategy moves precision@5 from 0.4 → 0.6 → 0.75. The LLM was never the problem.
Scenario 2 — “Users say the LLM is bad” Faithfulness score (RAGAS) = 0.65. The model is hallucinating into the chunks rather than from them. Fix: tighten the prompt with stricter grounding instructions (“Answer using ONLY the context below. Quote the source.”).
Scenario 3 — “Latency is fine, cost is exploding” Cost-per-answer rises linearly with traffic. The team is running the LLM on every retrieval, even on cache hits. Fix: add a Tier 0 cache (Redis) for popular questions with TTL 1-24h. Cached hits skip retrieval and LLM.