Retrieval-Augmented Generation (RAG) Architecture: Beyond the Basics

Almost every team building with large language models eventually builds a RAG system. The first version takes an afternoon: split documents into chunks, embed them, store the vectors, retrieve the top five matches for each question, and paste them into the prompt. It demos well. Then it meets real users, and the answers start drifting — confidently wrong, half-right, or correct but citing the wrong source.

The gap between a demo RAG pipeline and a production one is not a better embedding model or a bigger context window. It is advanced RAG architecture: how queries are transformed before retrieval, how candidates are gathered from more than one index, how they are re-ranked, how context is assembled, and how the system checks its own work. This article walks through the advanced patterns that close that gap — query decomposition, hypothetical document embeddings (HyDE), hybrid and recursive retrieval, and re-ranking — and explains when each one earns its added latency and cost.

If you already understand how modern LLMs are built and want to understand the retrieval layer that grounds them in your own data, this is the piece.

Why Naive RAG Breaks in Production

The original retrieval-augmented generation paper, published by Lewis et al. at Facebook AI Research in 2020, framed RAG as a way to combine a model’s parametric memory (what it learned in training) with non-parametric memory (an external index it can search). The idea was sound, and it remains the backbone of nearly every enterprise LLM application. But the simplest implementation — often called naive RAG — makes three assumptions that rarely hold.

Assumption one: the user’s question is a good search query. It usually isn’t. Users ask compound questions (“How did our refund policy change, and does it apply to EU customers?”), use vocabulary that doesn’t match the documents, or ask something whose answer is spread across several sources. A single embedding of the raw question captures an average of all of that intent and matches none of it precisely.

Assumption two: semantic similarity equals relevance. Dense embeddings are excellent at finding text that is about the same topic. They are much weaker at exact identifiers, product codes, rare names, negation, and numerical constraints. A passage that says “this policy does not apply to EU customers” can embed very close to one that says it does.

Assumption three: more context is better. Stuffing the top twenty chunks into a long context window feels safe, but research such as “Lost in the Middle” (Liu et al., 2023) showed that models use information at the start and end of a long context far more reliably than information buried in the middle. Irrelevant chunks are not neutral; they dilute attention and invite the model to blend sources.

Each of these failures shows up as a hallucination-like symptom, which is why teams often blame the generator. In practice, most RAG errors are retrieval errors: the right evidence never reached the model, or it arrived buried among distractors.

The Advanced RAG Pipeline: Five Stages, Not Two

Naive RAG has two stages: retrieve, then generate. A production pipeline separates the work into five stages, each with its own failure modes and its own levers.

Advanced RAG — From Question to Grounded Answer

User Question

1. Query Transformation decompose · rewrite · HyDE

Dense vector index Keyword index (BM25) 2. Hybrid retrieval → fuse (RRF)

3. Re-ranking cross-encoder · top 50 → top 5

4. Context Assembly dedupe · order · expand to parent · cite

5. Generation + Verification grounded answer · faithfulness check

Cited Answer or “not enough evidence” retry loop: weak evidence → reformulate query

Each stage can fail independently — and each can be measured independently.

The point of separating the stages is not complexity for its own sake. It is observability. When an answer is wrong, you can ask which stage failed: did the query transformation lose the user’s intent, did retrieval miss the document, did re-ranking push it out of the top five, or did the generator ignore it? Without that separation, every failure looks the same.

Stage 1: Query Transformation

The cheapest place to improve retrieval is before it happens. Query transformation uses an LLM call (often a small, fast model) to turn the user’s raw question into one or more queries that are better suited to the index.

Query Rewriting and Multi-Query Expansion

The simplest transformation rewrites a conversational question into a standalone search query. In a chat interface, “what about for contractors?” means nothing on its own; the rewriter folds in the conversation history to produce “contractor eligibility for the 2026 expense reimbursement policy.” Multi-query expansion goes a step further, generating three to five paraphrases of the question, running retrieval for each, and merging the results. Different phrasings hit different vocabulary in the corpus, which raises recall at the cost of extra retrieval calls.

Query Decomposition

Compound and multi-hop questions need to be split. “Which of our suppliers in Germany had delivery delays last quarter, and what penalties do their contracts specify?” contains at least two retrieval tasks, and the second depends on the answer to the first. Query decomposition asks a model to break the question into sub-questions, retrieves evidence for each, and either answers them sequentially (feeding earlier answers into later queries) or in parallel when they are independent.

A related technique, step-back prompting (Zheng et al., Google DeepMind, 2023), generates a more general version of the question first — “what are the penalty clauses in our standard supplier contract?” — and retrieves on both the specific and the general form. The general query often surfaces the principle or policy that the specific answer depends on.

Hypothetical Document Embeddings (HyDE)

HyDE, introduced by Gao et al. in the 2022 paper “Precise Zero-Shot Dense Retrieval without Relevance Labels,” addresses a structural mismatch: questions and answers do not look alike. A question is short and interrogative; the passage that answers it is longer, declarative, and uses domain vocabulary. Embedding the question and searching for nearby passages asks the embedding model to bridge that gap on its own.

HyDE bridges it explicitly. The system asks an LLM to write a hypothetical answer to the question — a plausible passage, which may contain factual errors — and embeds that passage instead of the question. Because the hypothetical answer is written in the register and vocabulary of real answers, its embedding lands closer to the genuine answer passages in vector space. The hypothetical text is never shown to the user; it is only a search key.

HyDE works best for knowledge-heavy questions over well-written corpora, and it is particularly useful when you have no labeled query–document pairs to fine-tune your retriever. It works poorly when the LLM knows nothing about the domain (its hypothetical answer points in the wrong direction) and adds a full generation step of latency before retrieval even starts. A common compromise is to run HyDE and direct-query retrieval in parallel and fuse the results.

Stage 2: Hybrid and Recursive Retrieval

Hybrid Search: Dense Plus Sparse

Dense retrieval (embeddings) and sparse retrieval (keyword methods such as BM25) fail in complementary ways. Dense search understands that “terminate the agreement” and “cancel the contract” mean the same thing; BM25 does not. BM25 reliably finds the passage containing “SKU-44817” or “Section 12.3(b)”; dense search often does not. Hybrid search runs both and merges the ranked lists.

The standard merging method is Reciprocal Rank Fusion (RRF), described by Cormack, Clarke, and Büttcher in 2009. Each document’s fused score is the sum of 1/(k + rank) across the lists it appears in, with k typically set around 60. RRF needs no score normalization — useful, because cosine similarities and BM25 scores live on incompatible scales — and it rewards documents that rank reasonably well in both lists over documents that rank first in one and nowhere in the other.

Chunking Is a Retrieval Decision

How you split documents determines what can be retrieved. Small chunks embed precisely but lose surrounding context; large chunks keep context but blur the embedding across several topics. Production systems increasingly decouple the unit of retrieval from the unit of generation:

• Small-to-big (parent-document) retrieval: index small chunks or individual sentences for precise matching, then return the larger parent section to the generator.
• Structure-aware chunking: split on headings, list items, table rows, and code blocks rather than fixed token counts, and prepend each chunk with its document title and section path so the embedding knows where it came from.
• Contextual chunk headers: use an LLM to write a one-sentence summary of how each chunk fits into its document, and embed that summary together with the chunk text.

Recursive and Hierarchical Retrieval

Some questions cannot be answered by any single chunk: “What are the recurring themes in last year’s customer complaints?” requires a view across hundreds of documents. Recursive retrieval builds an index at several levels of abstraction. RAPTOR (Sarthi et al., 2024) is a well-known example: it clusters chunks, writes a summary of each cluster, then clusters and summarizes those summaries, producing a tree. Retrieval can then match a broad question against high-level summaries and a narrow question against leaf chunks — or descend from a matching summary to the detailed passages beneath it.

A related approach, popularized by Microsoft’s GraphRAG work in 2024, extracts entities and relationships into a knowledge graph and summarizes communities of related entities. Graph-based retrieval shines for “connect the dots” questions that span many documents, at the cost of a substantial indexing pipeline that must be rebuilt as the corpus changes.

Stage 3: Re-Ranking Strategies

First-stage retrieval is optimized for recall: get the right document somewhere in the top 50 or 100, quickly, across millions of candidates. Re-ranking is optimized for precision: put the best five at the top. Splitting the job this way is what makes it affordable.

The workhorse re-ranker is a cross-encoder. Where an embedding model (a bi-encoder) encodes the query and each document separately and compares the vectors, a cross-encoder reads the query and a candidate document together in a single forward pass, letting attention flow between them. It can notice that a passage mentions the right product but the wrong year, or answers a different question with similar words. That joint reading is far more accurate — and far too slow to run over a whole corpus, which is why it only sees the shortlist.

Beyond pure relevance, a good re-ranking stage also handles diversity and business rules. Maximal Marginal Relevance (MMR) penalizes candidates that are near-duplicates of ones already selected, so the top five are not five copies of the same paragraph from different versions of a document. Metadata filters and boosts — prefer the current policy version, exclude documents the user is not authorized to see, favor sources marked authoritative — belong here too. Access control in particular must be enforced at retrieval time, never left to the generator.

Stage 4: Context Assembly

Once you know which passages to use, how you present them still matters. Good context assembly:

• Removes duplicates and near-duplicates that survived re-ranking.
• Orders passages deliberately — the strongest evidence first and, given the “lost in the middle” effect, important supporting evidence last rather than buried in the center.
• Labels every passage with a source identifier so the model can cite it and you can verify the citation.
• Keeps the context small. Five well-chosen passages routinely beat twenty mediocre ones, on accuracy as well as cost.

Long-context models have not made this stage obsolete. They have changed the trade-off: you can now afford to include whole sections rather than fragments, but the model’s ability to find and weigh evidence still degrades as you add irrelevant text.

Stage 5: Generation With Self-Checking

The final stage is where many teams stop investing, but several patterns make generation itself part of the retrieval loop.

Grounded Prompting and Abstention

Instruct the model to answer only from the provided passages, cite the passage behind each claim, and say plainly when the evidence is insufficient. Abstention — “I couldn’t find this in the documentation” — is a feature. A system that never abstains is a system that sometimes invents.

Corrective and Self-Reflective RAG

Self-RAG (Asai et al., 2023) trains a model to decide when to retrieve and to critique whether retrieved passages are relevant and whether its own output is supported by them. Corrective RAG (CRAG) (Yan et al., 2024) adds a lightweight evaluator that grades retrieved documents; if the evidence is judged weak, the system reformulates the query or falls back to another source before generating. You don’t need the specific trained models to borrow the pattern: a cheap grading step between retrieval and generation, with a retry path, catches a large share of failures before they reach the user.

Agentic Retrieval

The logical extension is to give the model retrieval as a tool and let it decide what to search for, inspect the results, and search again. This is how many AI agents handle research tasks. Agentic retrieval handles multi-hop questions gracefully but trades predictability for flexibility: latency and cost become variable, and you need limits on how many retrieval rounds a single question may consume.

A Systems View: RAG as a Set of Feedback Loops

It is tempting to treat a RAG pipeline as a linear conveyor belt. In production it behaves more like a system of interacting loops, and thinking about it that way explains several otherwise puzzling failures.

Two Loops That Govern a Production RAG System

Answer quality User trust Usage & feedback Fixes to corpus R trust loop

Corpus drift Retrieval errors Eval failures Re-indexing reduces (−) B evaluation loop

Without the balancing loop, corpus drift quietly erodes the reinforcing trust loop.

The first is a reinforcing loop: good answers build trust, trust drives usage, usage generates feedback that improves the corpus, and a better corpus produces better answers. The same loop runs in reverse just as easily — a few visible failures and users stop asking, which starves the system of the feedback it needs. The second is a balancing loop that has to be built deliberately: documents go stale, policies change, new products appear, and retrieval quality decays silently unless an evaluation process catches the drift and triggers re-indexing.

Teams that treat RAG as a one-time build tend to see quality peak at launch and erode afterward. Teams that build the balancing loop — scheduled evaluation, freshness metadata, ownership of the corpus — see it improve. That is the same lesson systems thinking brings to AI more broadly: behavior over time is shaped by loop structure, not by any single component.

Evaluating RAG: Measure Each Stage

You cannot improve what you only measure end to end. Useful RAG evaluation separates retrieval quality from generation quality:

• Retrieval metrics: recall@k (did the right passage appear in the top k?), precision@k, and mean reciprocal rank, measured against a set of questions with known supporting passages.
• Context metrics: context precision (how much of what you passed to the model was relevant) and context recall (whether it contained everything needed).
• Generation metrics: faithfulness (is every claim supported by the provided context?) and answer relevance (does it address the question?). The RAGAS framework (Es et al., 2023) popularized automated, LLM-judged versions of these.

Build a test set from real user questions, including the hard ones: multi-hop questions, questions with no answer in the corpus, and questions that hinge on an exact identifier. Run it on every change to chunking, embedding models, prompts, or re-rankers. Many “improvements” help one category while quietly breaking another.

Choosing the Right Patterns

Every technique above adds latency, cost, or operational complexity, and the best advanced RAG architecture is the simplest one that passes your evaluation set. A sensible order of adoption for most teams:

• Start with fundamentals: structure-aware chunking with titles and section paths, hybrid search with RRF, and a cross-encoder re-ranker. These three changes fix a large share of naive-RAG failures at modest cost.
• Add query rewriting as soon as you have a conversational interface.
• Add decomposition or HyDE when your evaluation set shows failures on compound questions or vocabulary mismatch, not before.
• Add recursive or graph retrieval only when users genuinely ask corpus-wide synthesis questions.
• Add corrective loops and agentic retrieval when correctness matters more than predictable latency.

Frequently Asked Questions

Is RAG still necessary now that models have very long context windows?

For most real corpora, yes. Even very long context windows hold a tiny fraction of a typical enterprise knowledge base, and filling them is slow and expensive on every request. Long contexts also do not remove the “lost in the middle” problem or the need for access control. What long context changes is the granularity of retrieval: you can retrieve whole sections or documents instead of small fragments, which makes chunking less fragile.

When should I fine-tune instead of using RAG?

Fine-tuning changes how a model behaves — its format, tone, or skill at a task. RAG changes what a model knows at answer time. Use RAG for facts that change, need citations, or must respect permissions. Use fine-tuning for consistent style, domain-specific reasoning patterns, or teaching a smaller model a narrow task. Many production systems use both, including fine-tuning the embedding model or re-ranker on in-domain query–document pairs.

Does HyDE make hallucinations worse, since it generates a fake answer?

No, as long as the hypothetical document is used only as a search key. It is embedded, used to find real passages, and then discarded; the final answer is generated from the retrieved passages, not from the hypothetical one. The real risk is a different one: if the model knows nothing about your domain, its hypothetical answer may point retrieval in the wrong direction. Running direct-query retrieval alongside HyDE and fusing the results guards against that.

How many chunks should I pass to the model?

There is no universal number, but most well-tuned systems pass a handful of high-quality passages — often three to eight — after re-ranking, rather than dozens. Tune it against your evaluation set: increase the count until context recall stops improving, and watch faithfulness and cost as you go.