The problem
Autoregressive inference keeps key and value activations for the tokens already processed. As a conversation grows, that cache becomes a first-class memory constraint—not an implementation detail.
The useful question is not simply how many tokens a model supports. It is how much of the current context can remain active in fast memory while the system continues to make progress.
What Flatline explores
Flatline defines an active KV-token limit and allocates separate K and V buffers for the model layers. When the active window fills, the prototype writes older cache pages to a local swap file, shifts the remaining active tokens, and records the page for later retrieval.
This is a research prototype, not a claim that SSD paging is free. Paging adds I/O and bookkeeping costs. It is useful because it makes the memory tradeoff explicit and gives the runtime a policy to reason about.
The engineering lesson
Long-context systems need an explicit cache policy: what stays hot, what can be paged, how pages are indexed, and when the runtime pays to bring history back. Treating KV memory as a managed resource makes those decisions visible to the product team.