Agentic RAG

Static RAG vs Agentic RAG

  STATIC RAG (Passive Pipeline):
  User Query ──► Vector Search ──► Single LLM Prompt ──► Final Output
  (If retrieval fails or is incomplete, generation fails!)

  AGENTIC RAG (Autonomous Decision Loop):
  User Query ──► Router Agent ──► [ Query Decomposition ]
                                        │
             ┌──────────────────────────┼──────────────────────────┐
             ▼                          ▼                          ▼
     [ Vector Search ]            [ SQL Database ]         [ Web Search API ]
     (Policy Documents)          (Sales Metrics)          (Live News Fallback)
             │                          │                          │
             └──────────────────────────┼──────────────────────────┘
                                        ▼
                             [ Evaluator Agent ]
                             "Is context sufficient?"
                             /                      \
                        Yes /                        \ No (Re-formulate Query)
                           ▼                          ▼
                   Final Response            Sub-Query Execution Loop

Core Agentic RAG Patterns

1. Corrective RAG (CRAG)

Evaluates retrieved document quality before passing to generation:

  Retrieved Context ──► [ Retrieval Evaluator Model ]
                              │
         ┌────────────────────┼────────────────────┐
         ▼                    ▼                    ▼
     CORRECT               AMBIGUOUS           INCORRECT
  Pass to LLM        Combine Internal +    Drop Internal; Trigger
                     Web Search Fallback   External Web Search

2. Self-RAG (Asai et al., 2023)

Fine-tunes the LLM to output explicit Reflection Tokens during generation:

• [Retrieve]: Triggers vector retrieval dynamically mid-generation.
• [IsRel]: Evaluates if retrieved text is relevant.
• [IsSup]: Evaluates if generation is supported by context (Factuality).
• [IsUse]: Evaluates overall answer utility.

3. Multi-Hop Query Decomposition

For complex queries ("Did company X or Y spend more on R&D in 2023?"):

1. Sub-Query 1: retrieve("Company X R&D spend 2023") $\implies $5\text{B}$.
2. Sub-Query 2: retrieve("Company Y R&D spend 2023") $\implies $8\text{B}$.
3. Synthesis: Compare values $\implies$ Company Y spent $$3\text{B}$ more.

Say this out loud

"Agentic RAG replaces static single-shot retrieval pipelines with an autonomous loop. It uses Dynamic Query Routing to direct queries to Vector DBs, SQL, or Web Search, query decomposition for multi-hop reasoning, and Corrective RAG (CRAG) to trigger web search fallbacks when internal vector retrieval confidence is low."

Follow-ups to expect

• What is Self-RAG Reflection Tokens? Special tokens ([Retrieve], [IsSupported], [IsRelevant]) generated by a fine-tuned LLM that allow the model to self-audit its generation and trigger vector lookups on demand.
• How do you prevent infinite loops in Agentic RAG? Set strict max_retries = 3 and max_steps bounds on the agent loop, defaulting to a graceful fallback answer if context remains insufficient.