Retrieval-augmented generation explained

This article explains one technique in depth: retrieval-augmented generation, usually shortened to RAG. It is the most common way to let a language model answer questions about a company’s documents, a product manual, or any information it was not trained on. The idea is simple; the quality depends on many small decisions.

The problem RAG solves

A language model has three gaps when it comes to your information:

• It never saw your internal documents, tickets, or databases.
• Its knowledge stops at its training cutoff.
• When it does not know something, it may invent a plausible answer.

Retraining the model every time a document changes is far too slow and expensive. RAG takes a different route: keep the documents outside the model, find the passages that matter for each question, and put them into the prompt. The model then answers from the text in front of it instead of from memory. The term comes from a 2020 research paper by Patrick Lewis and colleagues at Facebook AI Research.

How RAG works

A RAG system has two phases. Indexing prepares the documents once and again whenever they change. Querying runs for every question.

flowchart TB
  subgraph indexing["Indexing"]
    direction LR
    docs["Documents"] --> chunks["Split into chunks"]
    chunks --> embed1["Create embeddings"]
    embed1 --> store["Vector database"]
  end
  subgraph querying["Querying"]
    direction LR
    question["Question"] --> embed2["Create embedding"]
    embed2 --> search["Find similar chunks"]
    store -.-> search
    search --> prompt["Prompt with question and chunks"]
    prompt --> llm["Language model"]
    llm --> answer["Answer with sources"]
  end

Indexing

1. Load the documents: PDFs, web pages, wiki articles, tickets, database records.
2. Split them into chunks of a few hundred words, often with a small overlap so a sentence at the edge is not lost.
3. Embed each chunk. An embedding model turns the text into a list of numbers that represents its meaning.
4. Store the embeddings with the original text and metadata, such as the source, date, and access rights, in a vector database or a search engine with vector support.

Querying

1. Embed the question with the same embedding model.
2. Retrieve the chunks whose embeddings are closest to the question’s embedding. Similar meaning gives similar numbers, so “How do I get my money back?” finds a chunk about the refund policy even without the word “refund”.
3. Build a prompt that contains the instructions, the retrieved chunks, and the question.
4. Generate the answer, ideally with references to the chunks it used.

In code, the query step is short. The functions below stand in for an embedding API, a vector database, and a model API:

1
export async function answerQuestion(question: string): Promise<string> {
2
  const queryVector = await embed(question);
3
  const chunks = await vectorStore.search(queryVector, { limit: 5 });
4

5
  const sources = chunks
6
    .map((chunk, index) => `[${index + 1}] ${chunk.source}\n${chunk.text}`)
7
    .join('\n\n');
8

9
  return askModel(
10
    `Answer the question using only the sources below. Cite sources as [1], [2].
11
If the sources do not contain the answer, say so.
12

13
Sources:
14
${sources}
15

16
Question: ${question}`,
17
  );
18
}

The instruction to answer only from the sources, and to say when they are not enough, is what reduces invented answers.

What makes RAG work well

A RAG answer can only be as good as the retrieved text. If the right passage is not among the results, the model cannot use it. Most of the work in a RAG system goes into retrieval:

• Chunking. Chunks that are too small lose context; chunks that are too large dilute the match and waste tokens. Splitting along headings and sections usually works better than splitting at a fixed length.
• Hybrid search. Embeddings find meaning but can miss exact terms such as product codes, names, or error numbers. Combining vector search with classic keyword search covers both.
• Reranking. A first search returns a few dozen candidates quickly; a second, more precise model reorders them, and only the best go into the prompt.
• Metadata filters. Restricting the search by date, product, language, or department removes irrelevant matches before they compete.
• Access control. The search must respect the user’s permissions. Otherwise the model will quote a document the user was not allowed to see.
• Freshness. The index has to be updated when documents change or are deleted, or the system answers from outdated text.

Where RAG goes wrong

Typical failures, and where to look:

• The answer is missing or wrong, but the information exists. Retrieval did not find it. Check the chunking, try hybrid search, and test the search results on their own.
• The answer mixes two topics. Retrieved chunks from different products or versions were combined. Add metadata filters.
• The model adds facts that are not in the sources. Tighten the instruction, ask for citations, and check them automatically.
• Questions that need an overview fail. “What are the main complaints this year?” needs many documents, not the five most similar chunks. Such questions need summaries prepared in advance or an agent that searches several times.

RAG, fine-tuning, and long context

These approaches combine well. A fine-tuned model can use RAG, and a model with a large context window can receive more and longer retrieved passages.

Agentic RAG

In the classic design, the application always searches once before the model answers. In agentic RAG, search is a tool the model can call when it needs it. The model can rephrase the query, search several sources one after the other, read a full document after finding a promising chunk, and stop when it has enough. This handles complex questions better, at the cost of more model calls and less predictable behavior. Agentic AI describes the loop behind it.