Design: Enterprise RAG Chatbot

The Problem

Design a chatbot for an enterprise with 50,000 internal documents (HR policies, product manuals, engineering docs, legal contracts). Employees ask natural language questions and expect accurate, cited answers. The system must not hallucinate false information.

High-Level Architecture

  Employee Question: "What is the paternity leave policy?"
              │
              ▼
  ┌───────────────────────────────┐
  │ STEP 1: QUERY PROCESSING      │
  │ Rewrite conversational query  │
  │ into a standalone search query│
  └───────────────────────────────┘
              │
              ▼
  ┌───────────────────────────────┐
  │ STEP 2: HYBRID RETRIEVAL      │  (~20ms)
  │ BM25 + Dense Vector Search    │
  │ Output: Top 10 document chunks│
  └───────────────────────────────┘
              │
              ▼
  ┌───────────────────────────────┐
  │ STEP 3: RERANKING             │  (~100ms)
  │ Cross-Encoder reranker        │
  │ Output: Top 3-5 most relevant │
  └───────────────────────────────┘
              │
              ▼
  ┌───────────────────────────────┐
  │ STEP 4: LLM GENERATION        │  (~1-3s)
  │ Prompt = System Instructions  │
  │ + Retrieved Context + Question│
  │ Output: Grounded answer with  │
  │ citations [Source: doc.pdf p3] │
  └───────────────────────────────┘

Step 1: Document Ingestion Pipeline (Offline)

Before the chatbot can answer questions, all company documents must be processed:

1. Parse Documents: Extract text from PDFs, Word docs, HTML pages, Confluence wikis, and Slack threads. Handle tables, headers, and images.
2. Chunk Documents: Split long documents into overlapping chunks of 256 to 512 tokens. Overlap by 50 tokens so that ideas spanning chunk boundaries are not lost.
3. Embed Chunks: Run each chunk through an embedding model (like a sentence transformer) to produce a dense vector.
4. Index in Vector Database: Store chunk vectors in a vector database (like Qdrant or Pinecone) along with metadata (document title, page number, last updated date).
5. Build BM25 Index: Also index chunks in a keyword search engine (Elasticsearch) for exact term matching.

Step 2: Query Processing

When a user asks a follow-up question like "What about for adoptive parents?", the system does not know what "that" refers to without conversation history. Use an LLM to rewrite the query into a self-contained search query: "What is the paternity leave policy for adoptive parents?"

Step 3: Hybrid Retrieval and Reranking

Run BM25 and dense vector search in parallel. Merge results using Reciprocal Rank Fusion. Then pass the top 10 chunks through a Cross-Encoder reranker that reads each query-chunk pair together to produce a more accurate relevance score. Keep the top 3 to 5 chunks.

Step 4: LLM Generation with Grounding

Assemble a prompt with:

• System Instructions: "Answer the question using ONLY the provided context. If the context does not contain the answer, say 'I don't know'. Include citations referencing the source document and page number."
• Retrieved Context: The top 3 to 5 document chunks.
• User Question: The rewritten query.

The LLM generates an answer grounded in the retrieved documents, with inline citations like [Source: HR Policy v3, Page 12].

Key Design Challenges

1. Chunking Strategy: Too small and you lose context. Too large and you dilute relevance and waste context window tokens. Experiment with semantic chunking (split at paragraph or section boundaries) rather than fixed token counts.
2. Hallucination Guardrails: Even with retrieval, LLMs can still fabricate details. Add a post-generation verification step: check that key claims in the answer can be traced back to specific retrieved chunks.
3. Access Control: Employees should only see answers from documents they have permission to access. Apply metadata-based filtering at retrieval time.
4. Keeping Documents Fresh: Set up an incremental ingestion pipeline that re-processes updated documents and refreshes their embeddings and chunks.

Say this out loud

An enterprise RAG chatbot ingests documents by chunking and embedding them into a vector database. At query time, it rewrites conversational queries, retrieves relevant chunks via hybrid search, reranks them with a cross-encoder, and generates a grounded answer with citations using an LLM constrained to answer only from retrieved context.

Followups to expect

1. How do you evaluate RAG quality? Use metrics like answer faithfulness (does the answer match the retrieved context?), answer relevance (does it address the question?), and context precision (were the retrieved chunks actually useful?).
2. When would you fine-tune the LLM instead of using RAG? When the task requires a specific output format, tone, or reasoning style that prompting alone cannot achieve. RAG handles knowledge, fine-tuning handles behavior.