Skip to Content

LlamaIndex (RAG Framework)

When to use

Build a retrieval-augmented generation (RAG) pipeline where index documents, query them, and feed the top-K chunks into an LLM.

Analogy

A library catalogue. You index every book (parser + embeddings); a librarian (retriever) finds the right shelves; the assistant reads them and answers.

Data-flow diagram

   docs -> loader -> splitter -> embedder -> VectorIndex
                                                    |
                                                    v
                                       retriever(query) -> top-K chunks
                                                    |
                                                    v
                                       prompt(llm, query, chunks) -> answer

Deep explanation

LlamaIndex is the de-facto Python framework for RAG. It focuses on data plumbing: loaders, splitters, embedders, vector indexes, retrievers, query engines. The core abstraction is a QueryEngine that wraps retriever + prompt + LLM. Use it when retrieval-and-rewrite is the main loop; use raw OpenAI plus a vector store when you need very custom control.

Examples

Example 1

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
docs  = SimpleDirectoryReader('data').load_data()
index = VectorStoreIndex.from_documents(docs)
qe    = index.as_query_engine(similarity_top_k=3)
print(qe.query('What is the refund policy?'))

from_documents combines parser + splitter + embedder + index in one call — minimum viable RAG.

Example 2

# advanced: hybrid retriever (BM25 + vector)
from llama_index.core import Settings
from llama_index.retrievers.bm25 import BM25Retriever
vector  = index.as_retriever(similarity_top_k=3)
bm25    = BM25Retriever.from_defaults(documents=docs, similarity_top_k=3)
from llama_index.core.retrievers import QueryFusionRetriever
hybrid  = QueryFusionRetriever([vector, bm25], num_queries=4)
print(hybrid.retrieve('refund policy'))

QueryFusionRetriever fans out multiple queries and merges via RRF for hybrid BM25 + vector search.

Example 3

# customise the prompt that wraps retrieved chunks
from llama_index.core import PromptTemplate
qa_tmpl = PromptTemplate('Context: {context_str}\nQuestion: {query_str}\nAnswer:')
qe.update_prompts({'response_synthesizer:text_qa_template': qa_tmpl})

update_prompts lets you tune how the model formats answers without rebuilding the index.

Common mistake

Indexing without chunking. Embedding whole documents as one vector loses semantic granularity. Always chunk, then embed.

Key takeaway

LlamaIndex = load -> split -> embed -> index -> retrieve -> prompt -> answer. Tune chunk_size, similarity_top_k, and the response prompt for quality.

Production Failure Playbook

Failure scenario 1: no-chunking-bad-retrieval

Failure scenario 2: context-window-blowup