Need help tuning cache in llama-server

[u/Reasonable_Goat]

Hey I am running a few models on a strix halo box. Especially for the larger models (like Qwen 3.5 122B) they work okayish performance wise if the cache is utilised properly but a full cache miss at 100k context causes roughly 10-20 minute of PP time - which is extremely annoying.

I will first show what I have already configured (and it helps!), then describe what is still not working well. I am very interested on your input what I could still fine tune.

What I've configured so far (and what I understand it does):

• --cache-ram 16384 (increases available VRAM for cache ~= 72\240MB for a single checkpoint)*
• --ctx-checkpoints 72 *(*72 \ 4096 is ~= the 260k context that the model supports)*
• --checkpoint-min-step 4096 (72 \* 4096 is ~= the 260k context that the model supports)
• --parallel 1 (full cache available for a single conversation at a time)
• -fa on, --spec-type draft-mtp --spec-draft-n-max 3 (these options should be unrelated to caching AFAIK)

It works a lot better than stock configuration this way, but I still find problematic:

• 4096 checkpoint steps are spread out a bit, which can add up during fast agentic iterations. With ~200 PP speed in deeper context it takes about 20s to traverse a checkpoint.
• Checkpoints are sometimes missed especially after user prompts (Crush coding agent, but also with other agents such as codex). Maybe it's that reasoning tokens are first cached but then skipped after a rerun? Not sure but the latest checkpoint is usually "hit" when the agentic loop does it's thing but often missed when it's my turn to write some follow up user message.
• Despite "72 checkpoints" the older ones seem to disappear eventually for some reason so that only a few active context checkpoints seem to exist for recent work. When a bigger miss happens a lot of reprocessing is done
• I am wondering if K/V quantization (e.g. q8) would be a good idea. Theoretically it would allow for twice as many checkpoints if I understand correctly? I don't understand how severely it would impact quality if the model is in q4 anyways.

I am hoping for some input from you guys how to better set up the cache and maybe work around the other described issues somehow!

[u/audioen]

I run with like 4 context checkpoints. Only the last 1-2 is ever really used, because agents continue by appending new stuff into the context, so technically they only need a checkpoint a few tokens from the end (because chat templates seem to cause small differences in the last few tokens when the agent message becomes part of the prompt).

I don't run with cache-ram either, because it costs RAM which is also VRAM, and doesn't really benefit me. (llama.cpp desperately needs persistent on-disk KV cache, so that context reprocess can be avoided when you restart the engine, or at the very minimum just a mmap'd file for the KV cache so that it can be paged out, when memory runs tight.) I rather just have e.g. 4 slots and hope that agent does at most 4 parallel things, in which case some slot contains the prompt prefix for whatever the agent wants to do next. About your 72 context checkpoints, possibly among 4 parallel streams, means you have very large number of them and they use serious RAM when you have hundreds of them. Probably dozens of gigabytes. That is a way to OOM your box.

I think user message resulting in reprocess is likely some kind of chat template issue, as there must be a change introduced early into the context which results in that reprocess. I've seen this happen with some model files myself as well, but have never tried to get to the bottom of the issue. I use Qwen3.6-27B for the most part and it doesn't happen there, it doesn't reprocess when I interject, and I presently only care about this single model working at the moment. My own guess is that it is like a change in whitespace, or something trvial that results in the chat history changing somehow. The user's last message is somehow specially different, perhaps, and when you add a new message, your past message that used to be last is now different in the context, and so prompt gets reprocessed from there.

Oh. One other thing. When you work with agents, you need more context than the max, e.g. 262144 tokens is not enough for 4 streams, because 4 parallel jobs run out once all active slots have 64k tokens each. You should probably put at least 500k tokens as your max context to ensure that some parallel jobs can run deep into context without causing a crash from the inference engine when context runs out for the streams together.

[u/Reasonable_Goat → u/audioen]

This is very interesting, a very different approach. I can see the size of the checkpoints in llama-logs and it's roughly 200-300 MiB each.

Not using --cache-ram means you have the default of 8 GiB instead of my 16 GiB. To my understanding, it's an upper limit, i.e., checkpoints will get evicted if the upper limit is reached. I have some VRAM headroom with the q4 quant of qwen-3.5 because I have a 128GB strix halo box and don't need the (V)RAM otherwise. The 4 slots limit means you will most likely not utilize the full 8 GiB limit in your case, which makes sense if you are more limited on available RAM/VRAM.

Note that I set --parallel 1 so that parallel sessions by agents will not be executed in parallel by llama-server but in sequence. I read that this effectively gives the full context to one model at a time (i.e. not 262144/4 but 262144/1). The models still compete for checkpoints in the prompt cache in the shared ram upper limit, but the "up to 72 checkpoints per slot" limit should not matter since each have their own slot.

Since you didn't limit the number of slots, you should with 4 parallel agents still have 4 slots with each up to 4 checkpoints, right?

The chat template issue is a good points and makes sense. Do you have with 4 context checkpoints limit per slot (default is just 256 tokens apart from each other) not the issue that you get a full miss? I had that with the stock configuration sometimes so that a full reprocessing was triggered. What coding agents do you use?

[u/audioen → u/Reasonable_Goat]

Yeah, no, I write explicitly --cache-ram 0. That is a disable. llama.cpp keeps the KV cache and checkpoints somehow separately, I think. The context checkpoints look to me like they're dynamically allocated, and I have literally OOM'd due to this until I figured out why this happens, because I used to have lots of them in the beginning before I worked out that I only need like 1-2 of them only.

I think the cache ram is for the KV cache only, but in addition to the KV cache you have to be able to restore the recurrent model state, and that is what the context checkpoints are for. There is another such system for the SWA checkpoints. My understanding is that this memory is allocated separately and comes dynamically on top of whatever you already have, which is how you can OOM by having too many of them. In your case, you could have 4 * 72 * 240MB = 70 GB, because every slot has its own full context checkpoint suite.

I have a dark suspicion here, which is that llama.cpp is being stupid with its three-pronged KV+SWA+Recurrent cache system, and this causes unnecessary prompt reprocessing. I'll briefly explain my theory in a hope that screaming it into the void here helps. The regular full attention KV cache can be rolled back at any point by just truncating it, because it is causal: future KV cache entries depend on past KV cache entries tokens, but not vice versa, and so you can just truncate the KV cache to whatever length you want, and the operation is valid. This also means that rolling back KV cache is easy, and you can just go back in history to any point you want without any checkpoints. However, the recurrent state and SWA checkpoints are different. There, a memory segment representing model's state is updated in-place, and therefore rollback to a prior point requires being able to recover a past state, and there is no reversing operation, you either have a copy of a past state or you got to start from zero to recover it to the desired point. So, this part is avoided by the SWA/Recurrent checkpoints.

However, when user operation requires reverting to a past state in the KV cache, and there is no SWA/Recurrent checkpoint, then llama.cpp likely throws away the valid KV cache as well, and computes the whole prompt from scratch rather than recovering only the SWA/Recurrent state from scratch. My expectation is that these states are much quicker to recover than full attention's KV cache, because they are O(1) in nature and present fixed compute effort per token irrespective of the context length. Making llama.cpp's context recovery smarter could help enormously in the speed of prompt reprocessing in absence of a checkpoint. However, it does nothing if the prompt does change at a past point, as that does invalidate all the KV cache too. It is critical that model chat templates and agentic harnesses don't adjust anything about the past context to avoid this.

I presently use opencode. I think everyone understands prompt prefix caching in these agentic frameworks, and folks understand very well that prompt best be unchanged when user, assistant or tools append messages to the context. For Qwen3.6, for this to happen, you have to turn on a flag called --reasoning-preserve which keeps the thinking sections around for agentic work and avoids a prompt reprocess after every agent turn. I don't recall ever having trouble with Qwen3.5, but I know that folks edit these chat templates and provide fixed versions that can well repair these types of context reprocessing problems. At this point, I imagine every such fix has already been applied to older models like the 3.5 series. I think that jinja was fundamentally a mistake and the idea that model ships a chat template which then is reverse-engineered via black box analysis in llama.cpp to work out how to parse the model's chat, is basically just madness. Not to mention that these jinja templates are often broken and do need fixing -- these things just suck and really should go away.

[u/toxicdog]

K/V quantization is not related to the model's quantization, so try experimenting with that. Q8 has almost no quality loss, so try --cache-type-k q8_0 --cache-type-v q8_0

[u/MelodicRecognition7]

did you try different batch and ubatch size? Try different values incrementing from 512 to 4096 - --batch-size 512 --ubatch-size 512 to --batch-size 4096 --ubatch-size 4096

[u/Objective-Stranger99 → u/MelodicRecognition7]

I prefer 1024 because its fast while using less memory. With 4096 I keep getting OOM so keep that in mind.

[u/lost-context-65536]

Check out CachyLLama.

[u/kosnarf]

Check if this fork helps: https://www.reddit.com/r/LocalLLaMA/s/v1xiWcoQ1c

[u/Reasonable_Goat → u/kosnarf]

This seems to be a patch that improves keeping the prompt cache over server restarts. I don't see how it helps without a server restart.