Once a model can read a million tokens in one prompt, it is tempting to declare retrieval dead: just give it everything. A careful study says the answer is more interesting than that.
TL;DR
- Given enough resources, feeding the whole document to a long-context model tends to beat retrieval on answer quality.
- Retrieval (RAG) is dramatically cheaper because it sends far fewer tokens.
- A simple router that sends easy questions to RAG and hard ones to long context recovers most of the quality at a fraction of the cost.
Two ways to give a model context
There are two broad strategies for grounding an answer in your material. Retrieval-augmented generation (RAG) searches your data, pulls the few most relevant chunks, and puts only those in the prompt. Long context skips retrieval and puts the entire document (or many documents) into the window directly. As windows grew past a hundred thousand tokens, people started asking whether RAG was still worth the plumbing.
What the research shows
In Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach, researchers at Google DeepMind ran the two approaches head-to-head on the three strongest long-context models available at the time — Gemini-1.5-Pro, GPT-4o, and GPT-3.5-Turbo — across a suite of long-document QA datasets drawn from established benchmarks. For each question they compared two setups: feeding the whole document to the model (long context) versus retrieving the top-k chunks and feeding only those (RAG).
Their headline finding: when the model is strong and you can afford to feed it everything, long context consistently produces better answers than RAG on average. Retrieval sometimes drops the one chunk that actually held the answer; putting the full document in the window removes that failure mode. But the two approaches were also highly correlated — on the large majority of questions they returned the same answer, a fact the authors would go on to exploit.
RAG, by contrast, sends only a handful of retrieved passages, so it is far cheaper per query. The study frames this as a genuine trade-off rather than a winner: long context buys quality, RAG buys efficiency.
The hybrid: route, don't choose
That correlation is the opening. The authors propose Self-Route, a two-step pipeline that lets the model itself decide, per query, which path to take:
- RAG first, with an escape hatch. Give the model the retrieved chunks and ask it to either answer ordeclare the question "unanswerable" from what it was shown — no guessing.
- Escalate only the unanswerable. Questions the model flags as unanswerable are re-run with the full document in the window; everything else keeps the cheap RAG answer.
Because RAG and long context already agree on most questions, only a minority ever reach the expensive path. The result is answer quality close to always-long-context at a cost close to always-RAG.
The right question is rarely "RAG or long context?" — it is "which one does this particular query need?"
What it means for you
Two things follow for anyone building on top of an LLM. First, a bigger context window does not make retrieval pointless — it changes whenyou reach for it. Second, the smart default is not "always dump everything" but "keep the material organized so the system can pull the right slice, and only pay for the whole thing when a question truly needs it."
Where FocusLM fits
FocusLM keeps your material as a structured memoryinstead of one ever-growing prompt. That structure is what makes the cheap path possible: most questions are answered from the few relevant, cited files, and the broader context is there when a question genuinely needs it — the same route-don't-choose logic, applied to a memory that grows with you.