Should I use RAG or put everything in the context window?

What it is

This is a choice between two ways of giving a model your material. Full context hands it everything. RAG hands it only the passages a search step picks.

How it works

• Full context: attach the documents. Current flagship models from Anthropic, OpenAI and Google list context windows of about 1M tokens, so many business document sets fit (see tokens).
• RAG: index the library, retrieve the top passages for each question, and answer from those.
• The simple version: for a small codebase or document set, joining the files into one prompt is often enough. Simon Willison’s open-source files-to-prompt tool concatenates a directory of files into a single prompt for this.
• Rule of thumb: use full context when the material fits and the analysis is one-off. Use RAG when the library is too big, updates often, or serves many users.

Context sizes (checked 2026-10-17). Claude’s current models list 1M tokens. OpenAI’s GPT-6 models list 1.05M. Gemini 3.8 Flash accepts 1,048,576 input tokens. How many words that holds depends on each vendor’s tokenizer, so measure with your own documents.

Benefits

• Full context: no retrieval step can hide anything, and setup is quick. Andrej Karpathy has said that when the material fits, he wants all of it in the context window rather than behind a retrieval step. That’s his view, not a benchmark.
• RAG: scales to libraries of any size, stays current and can cite its passages.

Limits and risks

• Full context costs more per question, because every token is billed on every run. It can also suffer from context rot. A 2023 study (Liu et al.) found models used information in the middle of long inputs less well than at the start or end.
• RAG can miss the passage that mattered without telling you. Test it with questions whose answers you already know.