Advanced35 min

Retrieval-Augmented Generation (RAG)

Ground the model in your data by retrieving relevant context at query time.

By the end of this lesson, explain the RAG pipeline and why retrieval quality — not the model — usually decides answer quality.

How deep?

How the pieces actually move.

A language model only knows what was in its training data and what fits in its context window. RAG bridges that gap: at query time you retrieve relevant documents from your own knowledge base and place them in the context, so the model generates an answer grounded in current, private, or domain-specific data it was never trained on.

First, the whole system

First, see the whole system. Then we’ll open it up.

Index once, retrieve per query

Offline: documents are chunked, embedded, and stored in a vector DB.
Online: the query is embedded, similar chunks are retrieved, and the LLM generates an answer grounded in those chunks.

flowchart LR
  subgraph Index [Offline: index]
    docs[Documents] --> chunk[Chunk] --> emb[Embed] --> store[(Vector DB)]
  end
  subgraph Query [Online: answer]
    q[User query] --> qemb[Embed query] --> search[Similarity search] --> store
    store --> ctx[Top-k chunks] --> llm[LLM] --> ans[Grounded answer]
  end

Why it exists

You know what happens. Now see why it works.

It attacks hallucination and staleness without retraining. Instead of a model confidently inventing an answer, RAG gives it the actual source text to ground on — and lets you cite it. It's how you make an LLM useful over data that is private, changing, or too large to fit in any context window.

The pipeline has two phases. Indexing (offline): split documents into chunks, embed each chunk into a vector, and store it. Retrieval (online): embed the query, find the nearest chunks by vector similarity, and pass the top-k into the prompt. Chunking strategy, embedding model, and k are the main quality levers. Hybrid search (vector + keyword) and a re-ranking step often beat pure vector search.

Abstraction leak

This is where the abstraction starts leaking.

RAG is sold as 'give the model your docs' but it's really a search problem wearing an AI costume. If retrieval returns the wrong chunks, the model answers confidently from irrelevant context — garbage in, fluent garbage out. Most 'the LLM is wrong' complaints are actually retrieval failures.

Retrieval miss

Trigger: Poor chunking, weak embeddings, or too-small k means the right passage isn't retrieved.
Symptom: The model answers from irrelevant context or falls back to hallucinating.
Blast radius: Answer correctness and user trust.
Mitigation: Better chunking, hybrid search, re-ranking, retrieval evals, citing sources.

Deep dive: Data Inconsistency

Latency

RAG adds a retrieval hop (embedding + search + larger prompt) to every request, raising latency and token cost, in exchange for grounded, current, citable answers. Fine-tuning bakes knowledge in with lower per-query cost but can't stay fresh and is harder to attribute.

How it connects

If a RAG system gives a wrong answer, where should you look first?

At retrieval — whether the right chunks were actually fetched — since most RAG errors are retrieval failures, not generation failures.