A model's context window is a small, fixed desk. Operating systems solved "more data than fits in memory" decades ago — so what happens if you give an LLM the same trick?
TL;DR
- MemGPT treats the context window like RAM and adds a larger external store like disk.
- The model itself decides what to page in and out, so it can work beyond the window's limit.
- The lesson generalizes: managed, tiered memory beats one flat, ever-growing prompt.
The fixed-desk problem
Every model has a hard context limit. Long conversations, large documents, and accumulated history eventually overflow it, and once something falls out of the window the model behaves as if it never existed. Making the window bigger only pushes the wall further out — and, as the "lost in the middle" work shows, a bigger window is not evenly read anyway.
What the research shows
In MemGPT: Towards LLMs as Operating Systems, researchers at UC Berkeley borrow the oldest idea in systems design: a memory hierarchy. In an operating system, fast-but-small RAM is backed by slow-but-large disk, and the OS pages data between them so a program can address more memory than physically fits. MemGPT gives an LLM the same two tiers — a small main context inside the window and a large external context outside it — and lets the model manage the boundary itself through function calls.
Concretely, each tier is subdivided. Main context holds the fixed system instructions, a working-context scratchpad for facts the agent wants to keep in view, and a FIFO queue of recent messages. External context splits into recall storage — the full event log of past conversation — and archival storage, a searchable long-term store for arbitrary text. The model reads and writes across this boundary by emitting function calls, chained through a "heartbeat" so one action can trigger the next.
The trigger is a borrowed OS concept: an interrupt. When the window approaches its limit, a memory-pressure warning fires and the model evicts or summarizes the oldest messages into recall storage before continuing — the same "page out when RAM is full" move an operating system makes, but decided by the model.
Instead of trying to fit everything on the desk at once, the model learns to fetch what it needs and put back what it doesn't.
The authors tested this on the two workloads that break a fixed window. In multi-session chat, an agent had to remember and build on facts a user mentioned in earlier sessions — recalling them from external storage rather than losing them when they scrolled out of the window. In document analysis, MemGPT answered questions over documents far longer than the underlying model's context by paging relevant sections in on demand, including nested retrieval tasks designed to defeat a single pass. In both, the tiered design let a small window behave like a much larger one.
What it means for you
The practical takeaway is not "use MemGPT" specifically — it is the principle behind it. A useful assistant needs memory it can grow, organize, and selectively recall from, not a single prompt that you keep re-pasting and that silently forgets its own beginning. The question shifts from "how big is the window?" to "how good is the memory management?"
Where FocusLM fits
FocusLM is built on exactly this principle. Your material lives in a durable, structured memory outside any single conversation; each time you ask something, the relevant pieces are surfaced into context and cited, and everything else stays filed and out of the way. It is the memory-hierarchy idea applied to your real, growing body of knowledge — not just one chat session.