What Is RAG? Retrieval-Augmented Generation Explained

Retrieval-augmented generation (RAG) is a pattern where a system searches an external document index for relevant passages and passes those passages to a language model before it generates a reply. Instead of relying only on weights frozen during training, the model can answer using private, internal, or recently updated information.

A standard language model only knows what was in its training data up to its cutoff date. If a user asks about your company’s internal refund policy, a newly released API specification, or an account record updated five minutes ago, the base model has no way to know the answer from memory alone. RAG bridges that gap by pairing a search step with the generation step: retrieve the relevant facts first, then generate the response grounded in that retrieved text.

What Does Retrieval-Augmented Generation Mean?

Retrieval-augmented generation means combining an information retrieval system with a text generation model so the output is informed by external documents retrieved at query time. Rather than expecting a neural network to memorize every fact in its parameters, RAG treats an external text index as an open book the model can consult before speaking.

At a high level, every RAG workflow follows two core phases:

1. Retrieve: When a user submits a prompt, the system turns that query into a search request against an external data store—such as a vector database, a keyword search index, or a hybrid of both—and pulls back the most relevant text passages.
2. Generate: The system inserts those retrieved passages into the model’s context alongside the user’s original question, and the language model generates an answer based on the supplied passages.

Because the reference text is fetched dynamically per request, you can update what the system knows by updating the document index rather than retraining the underlying model. That makes RAG one of the standard patterns for building support assistants, internal documentation search, and domain-specific Q&A tools—alongside creative applications where teams use dedicated workflow tools like AI Studio Video for media production while relying on text APIs for knowledge retrieval.

How Did the 2020 RAG Paper Differ from Today’s RAG Pattern?

The term “retrieval-augmented generation” comes from a 2020 paper by Patrick Lewis and colleagues (arXiv:2005.11401), which combined a pre-trained sequence-to-sequence model with a dense vector index of Wikipedia, though today’s developers usually use the term for a simpler prompt-construction pipeline. Understanding both helps clarify why you will see “RAG” described in two slightly different ways across research papers and engineering docs.

The 2020 Lewis et al. Architecture

In the original 2020 paper and accompanying Meta AI research post, the authors framed language models as having two kinds of memory:

• Parametric memory: Knowledge stored inside the neural network’s weights—specifically a pre-trained BART sequence-to-sequence model.
• Non-parametric memory: Knowledge stored in an external dense vector index of Wikipedia articles, queried using a neural retriever called Dense Passage Retrieval (DPR).

In that research setup, the retriever and the generator were trained jointly end-to-end. The paper introduced two formulations for how retrieved passages influenced generation:

• RAG-Sequence: The model uses the same retrieved passages to generate the complete output sequence.
• RAG-Token: The model can draw on different retrieved passages for each individual token it generates.

On open-domain question answering benchmarks reported in the paper’s tables (arXiv:2005.11401), RAG-Sequence reached a 44.5 Exact Match score on the Natural Questions benchmark, outperforming both parametric-only seq2seq baselines and retrieve-and-extract architectures tested in the study.

Today’s Engineering Pattern

In modern application development, teams rarely train a retriever and generator jointly. Instead, “RAG” almost always refers to an inference-time pipeline built around off-the-shelf embedding models and general-purpose LLM APIs:

1. Split documents into smaller passages (chunks).
2. Convert those chunks into vector embeddings and store them in an index.
3. At query time, retrieve the top matching chunks.
4. Optionally rerank the results for better relevance.
5. Stuff the retrieved text chunks directly into the prompt and ask the LLM to answer.

Both the 2020 end-to-end neural architecture and today’s prompt-stuffing pipeline are called RAG. In day-to-day software engineering, unless someone specifically mentions training a retriever with a seq2seq model, they mean the modern pipeline pattern.

How Does a Modern RAG Pipeline Work Step by Step?

A production RAG system splits its work into two separate workflows: an offline ingestion pipeline that prepares and indexes your documents, and an online query pipeline that runs every time a user asks a question. Architecture guides such as AWS’s retrieval-augmented generation guidance and the AWS Generative AI Innovation Center RAG guide break these stages down into distinct components you can tune independently.

1. The Ingestion Pipeline (Indexing Data)

Before a user ever sends a prompt, your system needs to make your source documents searchable:

• Document loading and cleaning: Extract plain text and metadata (titles, URLs, timestamps, access permissions) from PDFs, wikis, databases, or markdown files, stripping out broken formatting or boilerplate chrome.
• Chunking: Break long documents into smaller segments so you only retrieve relevant passages instead of entire manuals. If chunks are too small, they lose surrounding context and cut off mid-thought; if chunks are too large, they pull in irrelevant noise and inflate your token usage. Teams often include a small overlap between adjacent chunks so sentences at the boundary do not lose their meaning.
• Embedding: Pass each chunk through an embedding model to turn its text into a dense numerical vector that represents its semantic meaning.
• Indexing: Store those vectors—alongside the raw text and metadata filters—inside a vector database or a hybrid search engine that pairs vector similarity with keyword matching (such as BM25).

2. The Query Pipeline (Answering Questions)

When a user asks a question at runtime, the query pipeline assembles the prompt:

• Query processing and embedding: Take the user’s input (and optionally rewrite or expand it for clarity) and convert it into a vector using the exact same embedding model used during ingestion.
• Retrieval: Search the index for chunks whose vectors are closest to the query vector, often combining semantic search with keyword matching and metadata filtering (for example, restricting results to documents the current user has permission to read).
• Optional reranking: Pass the initial candidate chunks through a cross-encoder reranker model, which scores how directly each passage answers the specific query and trims the list down to the strongest matches.
• Prompt assembly and generation: Insert the top-ranked passages into a structured prompt template alongside the user’s question, send the combined payload to the LLM, and return the generated answer (often with citations pointing back to the source metadata).

How Does RAG Compare to Long Context Windows and Fine-Tuning?

RAG, long context windows, and fine-tuning solve different parts of the model customization problem, and production systems frequently combine them rather than treating them as mutually exclusive choices. Choosing the right approach depends on whether you need dynamic facts, broad single-session comprehension, or changes in model behavior.

Dimension	Retrieval-Augmented Generation (RAG)	Long Context Window	Fine-Tuning
Primary Goal	Supply relevant facts from a large or changing corpus per query	Let the model read a large batch of text within one request	Change model style, tone, formatting, or specialized task behavior
How Knowledge Updates	Update or swap documents in the external index	Swap the raw documents pasted into the prompt	Collect new training examples and run another training job
Per-Request Token Impact	Low to moderate (only retrieved chunks are sent)	High (every document stays in the prompt payload)	Low (knowledge or behavior is baked into model weights)
Source Attribution	Straightforward to cite specific retrieved chunks	Possible, though harder to pinpoint across massive inputs	Difficult; weights do not track explicit document citations
Access Control	Enforced per query via index metadata filters	Must be filtered before assembling the prompt	Hard to isolate; anything in weights can leak to any user

RAG vs. Long Context Windows

With modern models supporting large context windows, it is tempting to skip retrieval and paste your entire knowledge base into every prompt. For a handful of short documents, that works fine—especially when paired with prompt caching and a warm KV cache so repeated prefixes do not have to be reprocessed from scratch.

However, once your corpus grows to thousands of files, dumping everything into the prompt hits two walls:

1. Cost and latency: Sending huge prompts on every turn drives up API spend and time-to-first-token, which is why selective retrieval remains one of the core strategies when looking at how to reduce AI API costs.
2. Attention degradation: Models do not attend equally well to every part of a massive prompt. As irrelevant text piles up in the window, accuracy often drops—a phenomenon known as context rot. RAG keeps the prompt focused by retrieving only the passages that matter for the current question.

RAG vs. Fine-Tuning

Fine-tuning updates a model’s weights on curated examples. It is effective when you want a model to follow a strict output schema, adopt a specific brand voice, or learn domain-specific syntax. It is a poor mechanism for teaching a model rapidly changing facts.

The 2020 Lewis et al. paper (arXiv:2005.11401) demonstrated why non-parametric memory matters for changing facts with a “hot-swap” index experiment. The authors tested their model using a December 2016 Wikipedia dump versus a December 2018 Wikipedia dump on questions about world leaders who had changed between those dates:

• When the query year matched the index year, the model answered 70% of 2016 world leader questions correctly using the 2016 index, and 68% of 2018 world leader questions correctly using the 2018 index.
• When the index was mismatched to the questions, accuracy dropped to 12% (2018 index on 2016 leaders) and 4% (2016 index on 2018 leaders).

Simply replacing the external index updated the system’s factual world knowledge without retraining a single model weight.

Where Does RAG Fail?

Adding retrieval to a language model reduces guesswork on private or recent topics, but it does not eliminate AI hallucinations or guarantee accurate answers. Because a RAG system has multiple moving parts, failures can happen in either the search step or the generation step:

• Missed retrieval: If the user phrases a question using different terminology than the source document—or if the chunking split a table away from its header—the retriever may fail to surface the passage that actually holds the answer. When the right chunk never reaches the prompt, the model is flying blind.
• Noisy or stale documents: An index is only as good as the files inside it. If your knowledge base contains both the 2023 and 2026 versions of an onboarding guide without clear deprecation filtering, the retriever will happily hand outdated instructions to the model.
• Over-retrieval: Fetching 25 chunks “just in case” often backfires. Packing the prompt with marginally related passages dilutes the actual answer, increases latency, and can trigger context rot where the model gets distracted by near-miss details.
• Indirect prompt injection via retrieved text: If your index ingests untrusted external content—such as incoming emails, public web pages, or user-submitted tickets—an attacker can plant adversarial instructions inside a document. When that document is retrieved and placed into the prompt, it can trigger a prompt injection attack that hijacks the model’s behavior.
• Model ignoring or misreading context: Even when the exact right passage is placed in the prompt, the generator can still misinterpret a negation, blend two separate chunks together inaccurately, or let its pre-trained priors override the provided text when the retrieved passage contradicts common knowledge.

Checklist for Building a Reliable RAG System

When moving a RAG prototype into production, a few disciplined engineering habits prevent the most common failure modes:

• Keep the index fresh and deduplicated: Build automated pipelines that remove deleted documents, archive superseded versions, and attach clear timestamps and source URLs to every indexed chunk.
• Retrieve less, but better: Rather than stuffing dozens of weak matches into the prompt, combine hybrid search (vector + keyword) with metadata filtering and a reranker so you only pass a small set of high-signal passages to the generator.
• Clearly separate untrusted retrieved text from system instructions: Wrap retrieved chunks in explicit structural delimiters (such as XML tags or labeled document blocks) and instruct the system prompt to treat those blocks strictly as passive reference data, never as executable commands.
• Tell the model to abstain when context is missing: Instruct the model to state clearly when the supplied passages do not contain enough information to answer the question. A prompt instruction cannot guarantee the model will always comply, but it gives the model a clearer fallback than silently guessing when retrieval comes up empty.

Frequently Asked Questions

Do you always need a vector database for RAG?

No. Retrieval-augmented generation only requires a way to fetch relevant text before calling the model. While vector databases are popular for semantic similarity search, many production RAG systems use traditional keyword search engines (like BM25 in Elasticsearch or PostgreSQL full-text search) or hybrid search that combines both keyword and vector matching.

Can RAG completely stop an LLM from hallucinating?

No. RAG reduces hallucinations caused by missing or outdated training data, because it gives the model the reference text directly in the prompt. However, the model can still misread a passage, combine facts from two unrelated chunks incorrectly, or guess an answer when the retriever pulls back irrelevant documents.

Should I use RAG or fine-tuning for my company’s internal docs?

Start with RAG when your goal is letting a model answer questions about internal documents that change over time. RAG lets you add, edit, or delete files in your index immediately and enforce per-user access permissions at query time, whereas fine-tuning is better suited for teaching consistent formatting, tone, or task workflows.

Sources

• Patrick Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — arXiv:2005.11401
• Meta AI Research Blog — Retrieval-Augmented Generation: Streamlining the creation of intelligent natural language processing models
• AWS Prescriptive Guidance — Understanding LLMs and RAG
• AWS Machine Learning Blog — From RAG to fabric: Lessons learned from building real-world RAGs at GenAIIC – Part 1