Retrieval augmented generation (RAG) solves a key limitation of language models: they only know what they learned during training – but they do not know your private documents, recent events, or domain-specific data. Retrieval-Augmented Generation (RAG) solves this by giving the model access to a knowledge base at inference time.
How Retrieval Augmented Generation Works
RAG has three steps: 1) Split documents into chunks and embed each chunk into a vector using a model like text-embedding-3-small. 2) Store the vectors in a vector database (like Chroma, Pinecone, or FAISS). 3) At query time, embed the user’s question, find the most similar chunks via cosine similarity, and prepend them to the prompt as context.
This approach was formalized by Lewis et al. (2020) in the seminal RAG paper, which showed that combining a retriever with a generator significantly improved knowledge-intensive tasks. Modern implementations use LangChain or LlamaIndex to orchestrate the pipeline.
Retrieval Augmented Generation: Chunking Strategies
The quality of a RAG system depends heavily on how documents are chunked. Common strategies include fixed-size chunks with overlap (256-512 tokens), sentence-based splitting, or semantic chunking using embeddings. Each chunk should be self-contained enough for the model to understand without surrounding context.
Run rag.py to see a complete RAG pipeline in action. The system indexes a set of documents, then answers questions by retrieving relevant passages and feeding them to the model as context.
In the next lesson, we will look at tool calling – how LLMs can use calculators, search engines, and APIs to augment their capabilities.
Retrieval Augmented Generation: Quality and Hybrid Search
The quality of a RAG system depends on retrieval precision. Pure vector search can miss documents where exact keyword matches matter (e.g., product codes, names). Hybrid search combines vector similarity with keyword (BM25) scoring for best results. Tools like Qdrant and Elasticsearch support hybrid search natively. The RAPTOR paper proposes building hierarchical document summaries for multi-hop reasoning, and Corrective RAG adds a verification step to filter out irrelevant retrieved passages before generation, improving answer accuracy by 5-10% on knowledge-intensive benchmarks.
Evaluation of RAG systems typically measures both retrieval recall (are the right documents found?) and generation faithfulness (does the answer stay true to the retrieved context?). The RAGAS framework provides standard metrics for both dimensions.
Advanced Retrieval Augmented Generation Patterns
Beyond simple retrieval, advanced RAG systems incorporate query rewriting – the model rephrases the user’s question before searching to improve retrieval quality. Self-RAG (Asai et al., 2023) adds a reflection step where the model decides whether retrieval is needed at all, and evaluates the relevance of retrieved passages. Multi-hop RAG chains multiple retrievals together: to answer “What is the population of the capital of France?”, the system first retrieves that Paris is the capital, then retrieves the population of Paris. The LangChain blog has excellent tutorials on implementing these patterns with their framework.
For production RAG systems, consider using purpose-built vector databases like Pinecone, Weaviate, or Chroma. These handle the embedding, indexing, and retrieval at scale with built-in filtering, hybrid search, and horizontal scaling.
RAG is the most widely adopted pattern for grounding LLMs in real data. By combining retrieval with generation, you get the best of both worlds: the linguistic fluency of a large language model with the accuracy and freshness of a curated knowledge base. This is why virtually every enterprise AI application uses some form of RAG.
Leave a Reply