Retrieval-augmented generation (RAG)

What it is

IBM describes retrieval-augmented generation as an architecture that connects an AI model to external knowledge bases. The term comes from a 2020 research paper by Lewis et al. Rather than loading a whole knowledge base into the context window, the system finds the few passages most likely to matter and passes those along.

How it works

1. Documents are split into chunks.
2. Each chunk is turned into an embedding, a numeric fingerprint of its meaning. Google, for example, offers Gemini Embedding models for semantic search and RAG.
3. When a question arrives, the system finds the chunks whose fingerprints are closest and adds them to the prompt.
4. The model answers from those chunks, ideally citing them.

Benefits

• Scale. Works on libraries far larger than any context window.
• Freshness. Update the library and the answers update, with no retraining. IBM contrasts this with fine-tuning, which trains domain knowledge into the model.
• Citations. Answers can point back to the exact source chunk.

Limits and risks

• Silent omissions. If retrieval misses the right chunk, the model never sees it, and nothing tells you. Andrej Karpathy has said he’d rather put all the material in the context window when it fits.
• Chunking is a design decision. Splitting documents badly breaks cross-references.
• More systems to run. An index, an embedding model and a retrieval service: each one is a place to fail.