A habit that saved me a lot of failed local runs: I started measuring my context budget before a run instead of discovering the ceiling when generation silently truncates or the KV cache OOMs my GPU.
The thing that finally clicked for me is that on local models the context window isn’t just a quality knob — it’s a hard memory bill you pay up front. KV-cache size scales with (context length × layers × heads × 2 × dtype bytes), so doubling the prompt you feed in can quietly double the VRAM you need for cache before a single token is generated. On a 24GB card that’s the difference between a run that fits and a run that pages/OOMs mid-generation.
What I do now, in order, before a long-context run:
- Tokenize the actual prompt, not a guess. The system prompt + retrieved chunks + few-shot examples + chat history almost always adds up to more than I expect. I count them with the model’s own tokenizer, because token/word ratios drift a lot between a code-heavy prompt and prose.
- Budget the reply too.
n_predict/max_tokensreserves cache. If I want a 1k-token answer I need room for prompt + 1k, not just the prompt. - Compare against the real ceiling I loaded with, not the model’s advertised max. If I loaded llama.cpp with
-c 8192, that 8192 is my wall regardless of what the model card says it can do — and rope-scaling to a bigger context has its own quality cost. - Trim at the retrieval/history layer, not by truncating the front of the prompt. Silent left-truncation is how you lose the system prompt and get a model that “forgot its instructions.”
The mindset shift: treat the context window like a resource you allocate deliberately, the same way you’d think about VRAM for weights. Once I could see the budget as a number before hitting enter, a whole class of “why did it cut off / why did it OOM / why did it ignore my instructions” problems just went away.
Curious what everyone else uses to keep an eye on this — do you eyeball token counts, script it, or just crank -c and hope the card holds?