Anyone else tried out KV cache blending?

Idea is simple-ish in abstract: instead of running normal prefill over all of a given prompt, split it into parts - generates caches for part A and part B in isolation, concatenate the result, feed it into decode like normal.

I honestly thought it'd totally fail. But I've been trying it out on Ling3-tiny for the non KDA layers - and if I give them some overlap between chunks, the model actually maintains full needle-in-haystack retrieval. Can even synthesise across the split parts. Currently running 256k prompts chunked into individual 4k bits and it seems to be unaffected on (simple) retrieval tasks so far.

There's probably a trade-off somewhere. But it's boosted my prefill speed by 3x, giving me ~1.3k tps prefill at 256k tokens - which is pretty much the same prefill speed as running Qwen3.8-27b on my 5090. So looks like it might be worth it