Retrieval-Augmented Generation

What is Retrieval-Augmented Generation (RAG)?

Standalone Large Language Models rely on Parametric Memory: facts stored inside their static parameter weights during pretraining.

Parametric memory has major flaws:

1. Knowledge Cutoff: Cannot answer questions about events after pretraining.
2. Private Data Blindness: Cannot access internal company documents or private databases.
3. Hallucinations: Generates plausible sounding but completely false facts when uncertain.

Retrieval-Augmented Generation (RAG - Lewis et al., 2020) solves this by connecting the LLM to Non-Parametric External Memory (a Vector Database).

  USER QUERY ──► [ EMBEDDING MODEL ] ──► Vector Query q
                                               │
                                               ▼
  [ VECTOR DATABASE ] ◄── Search Top-K Relevant Document Chunks!
         │
         ▼
  CONSTRUCT PROMPT: "Answer question based ONLY on context: <Chunks> Question: <User Query>"
         │
         ▼
  [ LLM GENERATOR ] ──► Grounded Answer with Source Citations!

Production RAG Architecture (2 Phases)

┌─────────────────────────────────────────────────────────────┐
│ PHASE 1: OFFLINE DATA INGESTION PIPELINE                    │
│ Raw PDFs / Docs ──► Text Chunking ──► Embedding Model       │
│ ──► Store Embeddings & Text Metadata in Vector Database!    │
└─────────────────────────────────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────┐
│ PHASE 2: ONLINE RETRIEVAL & GENERATION PIPELINE             │
│ User Query ──► Query Embedding ──► ANN Vector Search        │
│ ──► Top-K Chunks ──► Reranker ──► Prompt Context ──► LLM    │
└─────────────────────────────────────────────────────────────┘

Phase 1: Data Ingestion (Offline)

1. Document Parsing: Extract text from raw PDFs, Markdown files, or SQL tables.
2. Text Chunking: Split long documents into small overlapping passages (e.g. 512 tokens with 50-token overlap).
3. Vector Embedding: Pass text chunks through an Embedding Model (e.g. text-embedding-3-small or bge-large-en).
4. Vector Storage: Store embedding vectors alongside raw text metadata in a Vector DB (Qdrant, Pinecone, Milvus).

Phase 2: Online Query & Generation (Real-Time)

1. Query Embedding: Convert user query into a vector $\mathbf{q}$.
2. ANN Vector Search: Execute Approximate Nearest Neighbor search to retrieve top $K$ relevant chunks (e.g. $K=5$).
3. Prompt Construction: Format system prompt containing retrieved context passages.
4. LLM Generation: LLM reads context passages and synthesizes a grounded answer with source citations.

Key Benefits of RAG

1. Zero Hallucination Grounding: Forces the model to cite specific text chunks.
2. Real-Time Data Freshness: Updating enterprise data requires simply updating the Vector DB without retraining LLM weights.
3. Data Security & Privacy: Access control filters can be applied at the database level so users only retrieve documents they have permissions to see.

Say this out loud

RAG connects LLMs to external knowledge bases. It ingests documents by chunking, embedding, and indexing them in a vector database offline. During queries, RAG retrieves top-K relevant text passages using vector similarity and inserts them into prompt context, enabling grounded, hallucination-free generation with source citations.

Followups to expect

1. What is Naive RAG vs Advanced RAG vs Modular RAG? Naive RAG uses simple vector retrieval. Advanced RAG adds pre-retrieval query rewriting and post-retrieval reranking. Modular RAG incorporates dynamic graph search, routing, and agent loops.
2. What is RAG Triad Evaluation (Ragas)? Evaluating RAG quality across three metrics: Context Relevance (did retrieval find right chunks?), Groundedness (is LLM answer supported by context?), and Answer Relevance (does answer address user query?).