More context isn't always better: what happens past 64k tokens

The intuition is seductive: if some retrieved context helps, more must help more. Retrieve fifty documents instead of five. A large benchmarking study found the intuition breaks.

TL;DR

Adding more retrieved context helps — up to a point, then it plateaus.
For many models, accuracy actually declines once the context gets very long.
Only the strongest models hold up at very long context; most degrade earlier.

The "just retrieve more" temptation

In a retrieval system you choose how much to pull back. Since long windows are cheap to fill, why not retrieve generously and let the model sift? The hope is that more retrieved passages mean a higher chance the answer is somewhere in the prompt.

What the research shows

In Long Context RAG Performance of LLMs, researchers at Databricks ran retrieval experiments across 20 open-source and commercial models — spanning the major families — on three domain-specific datasets. For each, they held the retriever fixed and steadily grew the amount of retrieved context fed to the model, from 2,000 up to 128,000 tokens (and to two million where the model allowed it), measuring answer correctness at each step.

The pattern was consistent: accuracy improves as you add context at first, then saturates, and for many models declines as the context keeps growing. Crucially, only a handful of the newest state-of-the-art models held accuracy steady past roughly 64,000 tokens; most peaked earlier and fell off. More tokens in the window did not translate into more correct answers.

A recurring shape: accuracy climbs with retrieved context, plateaus once there is 'enough', then can dip as the prompt grows very long. Illustrative of the reported trend.

Just as telling were the ways models failed at long context. The study catalogued distinct failure modes that varied by model family — flatly wrong answers, refusing to answer (sometimes on safety or copyright grounds), ignoring the instructions, and lapsing into repeated or off-topic output. A long context does not fail in one clean way; it fails in several.

Filling the window is easy. Getting the model to use everything in it is the hard part — and past a point, adding more works against you.

This lines up with two things we have already seen: models under-read the middle of a long context, and long context trades cost for quality. Piling on retrieved text pushes more of the relevant signal into that under-read middle and pays for every extra token.

What it means for you

The practical rule is precision over volume. A few highly relevant, well-placed passages beat a giant dump of loosely related text — and cost less. If your system's answer quality plateaus, the fix is usually better selection, not more retrieved context.

Where FocusLM fits

FocusLM is built around precision. Because your material is filed as a structured memory rather than a flat pile, the system can surface the few passages that actually bear on your question — with citations — instead of flooding the model with everything and hoping. It is the same lesson the benchmark teaches, built into the product: the right context, not the most context.

Sources