Who wants to try a Pi trick for 27B to reuse prompt prefill between different sessions?

You know that when you start Pi you have to process the initial prompt, that takes some time when you use the slow dense QWEN 27B (that's the very reason why you use Pi instead of Cloud Code!), then you start to add extensions, tools, your append.md and whatever...
Well now that got big, like 20k big and it does bother.

So let's cache the "initial prompt" PP, so that when you start an new pi session: TA-DA! Instant ready, jolly good.

Well if you tried to do that with Pi, llama.cpp and QWEN 3.x hybrid KV all kind of things step in your way to prevent that, so many that I won't even start to count I'll just tell you what to do:

1. dwl and install this Pi extension (tested on Pi version 0.87.1 )
2. dwl and patch lama.cpp: yup no way around this if you kill the server between session, suck it or leave now
3. do your self a favor and use Froggeric template, for all your QWEN models, even the old ones.

----

So I'll help ya and give you some kinda useful parameters to launch the thing too:

--slot-save-path /home/eaman/llama/slot_caches/ --ctx-checkpoints 32 --checkpoint-min-step 4096 -np 1 --chat-template-file chat_template_3.8.jinja

You need to have save slots, that's the whole point, the caching is meant to resist restarts.
Beware the chunk of blocks cached follow ubatch boundaries, so yeah try to keep that down if you wanna cache some more.

Now in the extension README.md there's explanation of env variables that you can tweek, you go read those and edit accordingly to your setup (or have your LLM read that and suggest / config for you), TLDR you need at least:

 export PI_PREFIX_CACHE_BASE_URL=http://localhost:8080/v1 export PI_PREFIX_CACHE_PERSIST=1 export PI_PREFIX_CACHE_SLOT_DIR=/home/eaman/llama/slot_caches 

Disclaimer: this is not an easy thing, if you are not familiar with patching llama.cpp and installing extensions manually leave this thread for an other day.

On the other hand this thing kinda works for me so if someone else is interested after some testing (because all kind of evil things want to break prompt caching) I'll upload a final extension and see about the llama.cpp problem with saved check points..

Possible results:

https://preview.redd.it/xobe9ve7h5sh1.png?width=1281&format=png&auto=webp&s=129bd8cecbd83c2d137e2f8c4eb0a949665c2494