Long context or RAG? What the research says about very large context windows

Three findings worth knowing

Position matters. In "Lost in the Middle", Liu and colleagues studied how models use long inputs in tasks such as multi-document question answering. Performance was often highest when the relevant information appeared at the beginning or end of the context and dropped when it was in the middle¹. A later survey of long-context modelling describes this as a U-shaped curve⁵. The practical implication: where a fact sits in a long prompt can change whether it's used.

Advertised windows overstate useful windows. NVIDIA's RULER benchmark extended simple needle-in-a-haystack tests with harder tasks such as multi-hop tracing and aggregation. Models that scored almost perfectly on the simple test showed large drops as context length grew, and the authors report that many models fell below their quality threshold well before reaching their claimed context lengths². Finding one sentence in a haystack is a much easier job than reasoning across a long document.

Longer inputs degrade reliability. Chroma's 2025 "Context Rot" report tested 18 models, including several frontier models, on deliberately simple tasks, keeping difficulty constant while increasing input length. Performance grew increasingly unreliable as input grew, and models didn't use their context uniformly³. The authors note that real tasks are more complex, so the effect may be larger in practice, and they published their code for replication³⁴.

The other costs of stuffing the context

Quality isn't the only issue. Every token in the prompt costs money and adds latency on every request, unless the same prefix is cached and reused, which I cover in LLM inference serving. Long prompts are also harder to debug: when an answer is wrong, it is much harder to tell which of 200 pages misled the model than which of five retrieved passages did.

There's a governance cost too. Retrieval enforces access control naturally: you only fetch documents the user is allowed to see. "Put everything in the prompt" either ignores permissions or rebuilds them in a less tested way. For regulated data, that alone usually settles the question, as discussed in the HIPAA Security Rule proposal and UAE health data localisation.

When long context genuinely helps

None of this means long context is a gimmick. It changes what good retrieval looks like:

• Bigger, more coherent chunks. Instead of 300-token fragments, retrieve whole sections or whole documents, so the model sees definitions and exceptions alongside the rule.
• Single-document deep work. Reviewing one contract, one codebase module or one patient record against a checklist is a good fit, especially with the most important material placed at the start or end.
• Fewer retrieval misses. A wider net for recall, followed by a model that reads more candidates, can beat aggressive filtering that drops the right passage.
• Stable reference material. Long, unchanging context such as a policy manual or schema can be cached as a prefix and reused across requests.

How I design it: retrieve, then read generously

The common thread is what is now called context engineering: deciding deliberately what goes into the window, in what order, and what stays out. Retrieval, ranking, summarisation and memory are all tools for that decision. A large window gives you more room to be generous; it doesn't remove the need to choose.

Test it on your own data

Published benchmarks tell you the direction of the effect, not its size for your documents. Before choosing an approach, I run a simple comparison on the client's own evaluation set, as described in evaluation sets that predict production:

1. Baseline RAG with the current chunking and top-k.
2. RAG with larger chunks and more results, using the long window to read more.
3. Full-document long context for the subset of questions where it is feasible.
4. Position tests: the same answer-bearing passage placed at the start, middle and end of the context.

Measure accuracy, citation correctness, latency and cost per answer for each. The result is usually a hybrid, and the numbers make the case to stakeholders far better than any article, including this one.

The bottom line

Bigger context windows are a genuine improvement. They make retrieval more forgiving, enable new single-document workflows and reduce some engineering effort. But the evidence on position effects, effective context length and degradation with input size means the most reliable systems still choose what the model reads. Treat the window as a budget to spend well, not a bucket to fill.

How I can help

I design retrieval and context strategies for AI products, from chunking and ranking to long-context workflows, and test them against real data before they ship. See my AI architecture work and the RAG to production checklist, or analyse your challenge and bring the result to a call.