Inject external knowledge into the LLM’s prompt at inference time. The model sees documents it was never trained on.

Data Flow

QUESTION
    |
    v
+-------+       +-------------+       +-----------+
|embed  | ----> |vector store | ----> |top-k chunks|
+-------+       +-------------+       +-----------+
                                          |
                                          v
                          +---------------------------------+
                          | system + context(joined chunks) |
                          | + user question                 |
                          +---------------------------------+
                                          |
                                          v
                                       LLM
                                          |
                                          v
                                      ANSWER

What it is

A 3-stage pipeline:

1. Embed the question.
2. Retrieve top-k chunks from a vector store (often rerank).
3. Generate an answer conditioned on the retrieved context.

When to use

• Domain knowledge that changes (regulations, internal docs).
• Knowledge the base model has never seen.
• Reducing hallucinations on factual Q&A.

Code

from sentence_transformers import SentenceTransformer
import numpy as np
 
embedder = SentenceTransformer("all-MiniLM-L6-v2")
 
def retrieve(store, query, k=5):
    q = embedder.encode([query])[0]
    sims = store["emb"] @ q / (
        np.linalg.norm(store["emb"], axis=1) * np.linalg.norm(q) + 1e-12
    )
    top = np.argsort(-sims)[:k]
    return [store["text"][i] for i in top]
 
def generate(prompt, llm):
    return llm.complete(prompt + "\n\nUse the context above.")

Pitfalls

• Wrong chunk size: 5 chars = no signal, 50k chars = lost-in-the-middle.
• No reranking -> top-k by retrieval score is sometimes worst semantically.
• PII slipping into the corpus gets surfaced to the LLM — pre-redact.

Analogy

Open-book exam: you may not have memorised the answer, but you can look it up.

Interview tip: Chunk size + retrieval + reranking + grounded prompt — all four are needed.