This post was made with AI. I tried to remove as much slop as possible and keep it straight to the point to save your time as I know how annoying AI slop posts can be, but I still wanted to retain all the details so it can be used as a resource for comparison with other future quants, I advise that any humans just skim through it or read the 1st section and run the balanced profile with a context smart harness like pi or deepseek harness (apparently really good for this)

Optimizing Qwen3.6 / Qwen3.8 27B on 16GB VRAM: Complete Experimental Log, Benchmarks, and Deployment Guide

This document records the complete set of benchmarks, quantization evaluations, KV cache sweeps, speculative decoding experiments, and context scaling tests conducted across multiple GPU architectures to determine the optimal configuration for running Qwen 27B hybrid models within a 16GB VRAM constraint.

1. Primary Recommendations (Quick Reference)

1.1 Balanced Profile (Recommended Default)

• Model: Qwen3.8-27B-IQ4_XS-pure-MTP.gguf (14.56 GB)
• Inference Engine: beellama.cpp
• KV Cache Format: kvarn4 (4-bit variance-normalized KVarN)
• Speculative Decoding: Native MTP at draft depth 2 (--spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.0)
• Maximum Context (100% VRAM): 32,768 to 48,000 tokens
• Quality Metric: 92.55% Top-1 token match, 0.1169 mean KL divergence vs. uncompressed Q8_0 reference

1.2 Extended Context Profile (For Context > 48K)

• Model: Qwen3.8-27B-AD-IQ3_S-IQ3_XXS.gguf (12.98 GB)
• KV Cache Format: kvarn4
• Speculative Decoding: Native MTP at draft depth 2
• Maximum Context (100% VRAM): ~72,000 tokens (+50% context headroom)
• Quality Metric: 89.85% Top-1 token match, 0.2282 mean KL divergence vs. uncompressed Q8_0 reference

1.3 Recommended Server Launch Command

export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH ./build/bin/llama-server \ -m ./models/Qwen3.8-27B-IQ4_XS-pure-MTP.gguf \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ --spec-draft-p-min 0.0 \ --cache-type-k kvarn4 \ --cache-type-v kvarn4 \ -c 32768 \ -ngl 999 \ -fa on \ -np 1 \ --fit off \ -t 8 \ --host 127.0.0.1 \ --port 8080 \ --api-key your-key-here 

2. Quantization Quality Benchmarks

2.1 Final Evaluation vs. Q8_0 Gold Reference

Evaluation conducted using llama-perplexity against a gold reference generated from uncompressed Qwen3.8-27B-Q8_0.gguf (29.05 GB, PPL 3.9616).

• Corpus: 20 randomly sampled reasoning and math problems from canada-quant/hy3-w4a16-mtp-calibration (51,831 bytes, 24 chunks of 512 tokens).
• KV Cache: kvarn3 throughout to maintain exact numerical consistency.
• Logits Reference: Published on HuggingFace Hub as MaxDevv/Qwen3.8-27b-Q8-Logits-for-kld (3.04 GB).

Model Quantization	File Size	PPL (hy3, 512 ctx)	Mean KLD vs. Q8 (Lower = Better)	Top-1 Match vs. Q8 (Higher = Better)	RMS Δp	Notes
Q8_0 (Gold Baseline)	29.05 GB	3.9616 ± 0.131	0.0000	100.0%	—	Uncompressed reference baseline
Qwen3.8 IQ4_XS-pure	14.56 GB	4.1474 ± 0.142	0.1169 ± 0.011	92.55 ± 0.34%	8.87%	Highest distribution fidelity; recommended
Atomic AD-IQ3_S-IQ3_XXS	12.98 GB	3.9594 ± 0.129*	0.2282 ± 0.014	89.85 ± 0.39%	12.87%	Best sub-4-bit quant; recommended for high context
Atomic AD-IQ3_XXS	12.08 GB	4.8501 ± 0.183	0.3100 ± 0.018	87.93 ± 0.42%	13.91%	Plain 3-bit dynamic; moderate degradation
Ridge 3.7bpw	12.60 GB	4.5645 ± 0.163	0.3554 ± 0.019	86.44 ± 0.44%	15.16%	Fixed 3.7bpw; notable distribution divergence
Qwen3.6 IQ3_XXS-MTP	12.20 GB	4.6392 ± 0.185	0.9034 ± 0.035	78.79 ± 0.52%	22.99%	Evaluated on Qwen3.6 weights; severe quality drop

\Note on Atomic PPL: The* hy3 dataset serves as the calibration source for the Atomic/w4a16 quantization pipeline. This artificially flatters its raw PPL score on this corpus. The KLD (0.2282) and Top-1 match (89.85%) reflect its true, unbiased generalization accuracy.

2.2 Preliminary Wikitext-2 Quality Benchmarks

Earlier exploratory testing evaluated quants against a 4-bit IQ4_XS-pure local reference over 20 chunks of Wikitext-2 (512 context):

Model Quantization	File Size	Perplexity (Wikitext-2)	Mean KLD vs. 4-bit Ref	Evaluation
Qwen3.8 IQ4_XS-pure	14.56 GB	7.1252 ± 0.249	0.0000 (Ref)	Baseline 4-bit reference
Atomic AD-IQ3_S-IQ3_XXS	12.98 GB	7.3215 ± 0.256	0.0567	Retains high linguistic consistency
Ridge 3.7bpw	12.60 GB	7.6710 ± 0.267	0.1373	+7.7% PPL degradation vs. 4-bit
Qwen3.6 IQ3_XXS-MTP	12.20 GB	7.8892 ± 0.296	0.3552	Demonstrates clear sub-3.5bpw degradation

2.3 External Baseline Reference: Comprehensive Evaluation of Qwen3.6-27B Quantizations

To contextualize experimental results against baselines, the table below compiles the empirical benchmark dataset published by Quesma (Piotr Migdał, Do Qwen3.6 27B quantizations break the pelican?, July 2026).

This dataset tracks statistical distribution drift, downstream mathematical reasoning, terminal agent execution, generative SVG visual fidelity, and M5 Max execution throughput across 24 distinct quantization formats derived from Qwen3.6-27B-BF16.

Master Quantization Reference Table

Quantization Format	Family	Size (GB)	Mean KLD vs. BF16	Top-1 Match (Temp 0)	Token Collision (Temp 1)	Wikitext-2 PPL	AIME-120 Solved (%)	Terminal-Bench 2.1 Passed (%)	Pelican Bradley-Terry Score	Generation Speed (M5 Max)
BF16 (Reference)	Uncompressed	54.7	0.0000	100.0%	48.48%	6.40	70.8% (85/120)	38.2% (34/89)	+1.76 ± 9.05	24.9 tok/s
UD-Q8_K_XL	Unsloth Dynamic	35.8	0.00038	99.4%	48.47%	6.39	70.0% ($p=1.00$)	—	+1.27 ± 3.53	31.8 tok/s
FP8	vLLM Float	30.9	0.01700	96.7%	—	—	—	—	-0.21 ± 1.90	—
Q8_0	Standard K	29.0	0.00049	99.3%	48.46%	6.39	67.5% ($p=0.45$)	36.0% ($p=0.75$)	+1.19 ± 2.09	30.8 tok/s
UD-Q6_K_XL	Unsloth Dynamic	26.0	0.00140	98.8%	48.44%	6.40	65.8% ($p=0.18$)	—	+1.70 ± 1.07	33.1 tok/s
NVFP4 (Unsloth)	vLLM Float	23.3	0.04400	92.2%	—	—	—	—	+0.27 ± 0.81	—
Q6_K	Standard K	22.9	0.00210	98.3%	48.46%	6.41	64.2% ($p=0.12$)	30.3% ($p=0.17$)	+2.54 ± 2.59	33.0 tok/s
NVFP4 (NVIDIA)	vLLM Float	21.9	0.03900	92.1%	—	—	—	—	+1.77 ± 2.63	—
UD-Q5_K_XL	Unsloth Dynamic	20.4	0.00460	97.4%	48.41%	6.40	70.8% ($p=1.00$)	—	+1.39 ± 1.02	30.3 tok/s
Q5_K_M	Standard K	19.8	0.00520	97.2%	48.38%	6.40	65.0% ($p=0.12$)	29.2% ($p=0.13$)	+1.12 ± 0.88	29.5 tok/s
Q5_K_S	Standard K	19.3	0.00580	97.1%	48.39%	6.40	67.5% ($p=0.52$)	—	+1.59 ± 0.80	30.4 tok/s
UD-Q4_K_XL	Unsloth Dynamic	17.9	0.01300	95.8%	48.29%	6.45	70.8% ($p=1.00$)	—	+0.48 ± 0.85	32.8 tok/s
Q4_1	Legacy Q4	17.5	0.01800	94.8%	48.19%	6.43	64.2% ($p=0.14$)	—	+0.82 ± 4.09	34.4 tok/s
Q4_K_M	Standard K	17.1	0.01700	95.1%	48.29%	6.45	65.8% ($p=0.21$)	40.4% ($p=0.79$)	+1.09 ± 0.90	32.7 tok/s
IQ4_NL	Importance Matrix	16.3	0.01800	94.8%	48.19%	6.45	66.7% ($p=0.33$)	—	+2.54 ± 6.36	32.7 tok/s
Q4_0	Legacy Q4	16.1	0.03500	93.0%	48.10%	6.55	64.2% ($p=0.15$)	—	+1.31 ± 0.65	34.7 tok/s
Q4_K_S	Standard K	16.1	0.01900	94.7%	48.23%	6.46	63.3% ($p=0.06$)	—	+2.18 ± 0.98	33.5 tok/s
IQ4_XS	Importance Matrix	15.7	0.01800	94.7%	48.20%	6.48	66.7% ($p=0.27$)	—	+2.32 ± 1.80	27.4 tok/s
UD-Q3_K_XL	Unsloth Dynamic	14.8	0.03800	92.5%	48.16%	6.51	72.5% ($p=0.81$)	31.5% ($p=0.24$)	+0.46 ± 1.87	28.7 tok/s
Q3_K_M	Standard K	13.8	0.05000	91.0%	48.10%	6.50	73.3% ($p=0.65$)	—	+0.80 ± 0.76	32.1 tok/s
Q3_K_S	Standard K	12.6	0.08300	88.3%	47.49%	6.71	54.2% ($p<0.001$)	—	+0.99 ± 1.88	31.3 tok/s
UD-IQ3_XXS	Unsloth Dynamic	12.2	0.08600	88.2%	47.02%	6.82	52.5% ($p<0.001$)	—	-0.32 ± 2.66	26.0 tok/s
UD-Q2_K_XL	Unsloth Dynamic	12.0	0.12000	86.5%	45.73%	6.84	56.7% ($p<0.001$)	32.6% ($p=0.30$)	-1.25 ± 2.50	30.9 tok/s
UD-IQ2_M	Unsloth Dynamic	11.0	0.13000	85.5%	45.82%	7.03	54.2% ($p<0.001$)	—	-0.61 ± 2.13	26.6 tok/s
UD-IQ2_XXS	Unsloth Dynamic	9.6	0.28000	78.7%	43.53%	8.19	32.5% ($p<0.001$)	—	-3.29 ± 1.95	27.8 tok/s

Metric Definitions and Key Insights

1. Statistical Collapse Threshold ($\text{KLD} \le 0.050$ vs. $\text{KLD} > 0.080$):
   • Quantizations maintaining $\text{KLD} \le 0.050$ (Q4_K_M, UD-Q4_K_XL, Q3_K_M, UD-Q3_K_XL) exhibit no statistically significant degradation on complex reasoning (AIME-120 accuracy matches BF16 within standard error margins).
   • Once $\text{KLD}$ exceeds $0.080$ (Q3_K_S, UD-IQ3_XXS, and all 2-bit formats), reasoning performance drops sharply ($p < 0.001$). On AIME-120, accuracy drops from ~70–73% down to 32–56%.
2. Top-1 Agreement (Greedy Decoding Fidelity):
   • Measures token-level exact match against the unquantized BF16 model at $T = 0$.
   • Q8_0 reaches 99.3%, 4-bit quants achieve 93.0%–95.8%, and 3-bit quants achieve 88.2%–92.5%.
   • Below 3-bit, agreement degrades to 78.7% (UD-IQ2_XXS), causing severe output drift.
3. Collision Cross-Entropy (Sampling Consistency at $T = 1.0$):
   • Measures the probability that independent random samples from the quantized model collide with samples from BF16.
   • The unquantized baseline BF16 compared against itself has an inherent collision ceiling of 48.48%.
   • Standard 8-bit to 4-bit models stay tightly clustered between 48.10% and 48.47%. Degradation only becomes pronounced below 3 bits (dropping to 43.53% on UD-IQ2_XXS).
4. Visual Spatial Reasoning (Pelican Bradley-Terry Elo):
   • Assessed via blind pairwise generation duels of complex vector graphics ("a pelican riding a bicycle" and multi-gear train animation scripts) judged by Gemini 3.6 Flash.
   • Models $\ge \text{4-bit}$ show score parity dominated by generation seed variance.
   • 2-bit models (UD-IQ2_XXS at $-3.29$) fail consistently due to SVG syntax errors, malformed coordinate paths, and dropped rendering tags.

2.4 Technical Note on IQ4_XS-pure

Standard llama.cpp builds hardcode attention projection tensors (attn_qkv) to Q5_K (via commit 1dab5f5a), inflating file size from 14.6 GB to 15.1 GB without measurable quality benefit.

• Standard IQ4_XS: 15.1 GB (Exceeds safe limits at 32K context on 16GB cards).
• IQ4_XS-pure: 14.56 GB (Identical PPL: 7.3804 vs 7.3765, frees ~550 MB VRAM).

3. KV Cache Compression Benchmarks

Tested on an RTX 5060 Ti 16GB with single-stream decode at 32,768 fully-filled context tokens using Qwen3.6/3.8 IQ4_XS-pure with MTP draft depth 2:

KV Cache Type	Implementation	Bits / Value	Compression	Decode Speed (@32K filled)	PPL vs. FP16 (6.5407)	Quality Impact
f16	Reference	16.0	1.0×	— (OOM at 32K)	6.5407	Full precision
turbo2	TheTom	2.25	6.4×	40.86 tok/s	KLD 0.0056	Noticeable distribution drift
kvarn5	beellama	5.0	3.2×	38.33 tok/s	~Lossless	Minor throughput improvement
kvarn4	beellama	4.0	4.0×	38.10 tok/s	6.5523 (+0.2%)	Optimal throughput and quality
kvarn3	beellama	3.0	5.3×	36.71 tok/s	6.5415 (= FP16)	Mathematically lossless
turbo4	TheTom	4.25	3.8×	34.69 tok/s	~Lossless	Slower decode execution
turbo3	TheTom	3.25	4.9×	33.76 tok/s	~Lossless	Slower decode execution
q8_0	Stock llama.cpp	8.0	2.0×	21.30 tok/s	~Lossless	High memory bandwidth overhead

4. Speculative Decoding Sweep: MTP vs. dflash vs. ngram

4.1 Multi-Token Prediction (MTP) Tuning

MTP uses the built-in draft prediction head embedded in the base model GGUF.

Tested with turbo3 KV cache at 32,768 filled context:

Draft Length ($n$-max)	Minimum Probability ($p$-min)	Decode Speed (@32K filled)	Acceptance Ratio	Notes
$n = 1$	$p = 0.0$	16.40 tok/s	1.00	Autoregressive baseline equivalent
$n = 2$	$p = 0.0$	22.10 tok/s	0.79	Optimal balance of speed and stability
$n = 3$	$p = 0.0$	13.60 tok/s	0.52	Rejection penalty degrades throughput
$n = 4$	$p = 0.0$	21.40 tok/s	0.68	High variance across token sequences
$n = 4$	$p = 0.5$	6.40 tok/s	0.21	Early truncation causes execution stalls

4.2 dflash (Block Diffusion) Speculative Decoding

Evaluated using standalone diffusion draft models (dflash-Qwen3.6-27B) across various draft quantizations and draft depths:

Target Quant	Draft Model Quant	Draft Length ($n$)	Target Hardware	Context Length	Decode Speed	Evaluation
UD-IQ3_XXS	dflash-Q4_0 (0.98 GB)	$n = 4$	4060 Ti	512 / 512	50.06 tok/s	Fast, draft fits comfortably
UD-IQ3_XXS	dflash-Q4_0 (0.98 GB)	$n = 8$	4060 Ti	512 / 512	43.90 tok/s	Diminishing acceptance at $n=8$
UD-IQ3_XXS	dflash-Q8_0 (1.85 GB)	$n = 4$	4060 Ti	512 / 512	54.85 tok/s	Highest peak speed; draft tight on VRAM
UD-IQ3_XXS	dflash-Q8_0 (1.85 GB)	$n = 8$	4060 Ti	512 / 512	OOM	CUDA compute buffer exceeds 16GB
UD-IQ3_XXS	dflash-Q4_0 (0.98 GB)	$n = 8$	T4 (16GB)	512 / 512	24.20 tok/s	Max speed achieved on T4
Q3_K_S	dflash-Q4_0 (0.98 GB)	$n = 4$	T4 (16GB)	512 / 512	18.64 tok/s	Limited by compute bounds
UD-IQ3_XXS	dflash-Q4_0 (0.98 GB)	$n = 8$	5060 Ti	16K Cap	31.42 tok/s	Low acceptance ($0.35$) at 16K

Comparison Summary: While dflash achieves high peak throughput on short contexts with small target models (up to 54.8 tok/s), it requires 1.0–1.85 GB of dedicated VRAM for the draft model. Native MTP requires only ~486 MB of draft context, delivers consistent acceptance rates at long contexts, and allows larger base models (IQ4_XS) to fit on 16GB hardware.

4.3 Draft-less n-gram Speculative Decoding (ngram-mod)

• Flags: --spec-type ngram-mod --spec-ngram-mod-n-min 16 --spec-ngram-mod-n-max 32 --spec-ngram-mod-n-match 24
• VRAM Overhead: 0 MiB GPU memory.
• Performance: Provides large throughput acceleration (+35 to +135 tok/s) when generating repetitive structures or editing text present in context (e.g., refactoring code). Yields zero benefit on novel generation.

5. Multi-System Cross-Hardware Benchmarks

Single-stream decode performance (batch size 1, 512 prompt tokens / 512 generated tokens) across tested hardware platforms:

Hardware Platform	Architecture	Compute Cap	Memory Bandwidth	Quantization Model	Speculative Configuration	Decode Throughput
Modal Tesla T4	Turing	sm_75	320 GB/s	bartowski Q3_K_M	None	8.70 tok/s
Modal Tesla T4	Turing	sm_75	320 GB/s	unsloth Q3_K_M-MTP	None	10.29 tok/s
Modal Tesla T4	Turing	sm_75	320 GB/s	unsloth UD-Q3_K_XL	None	10.33 tok/s
Modal Tesla T4	Turing	sm_75	320 GB/s	Q3_K_M-MTP	MTP $n = 2$	16.12 tok/s
Modal Tesla T4	Turing	sm_75	320 GB/s	UD-IQ3_XXS	dflash-Q4 ($n = 8$)	24.20 tok/s
RTX 4060 Ti 16GB	Ada	sm_89	288 GB/s	Q3_K_M-MTP	None	19.12 tok/s
RTX 4060 Ti 16GB	Ada	sm_89	288 GB/s	Q3_K_M-MTP	MTP $n = 2$	34.31 tok/s
RTX 4060 Ti 16GB	Ada	sm_89	288 GB/s	UD-IQ3_XXS	dflash-Q8 ($n = 4$)	54.85 tok/s
RTX 5060 Ti 16GB	Blackwell	sm_120	448 GB/s	Q3_K_M-MTP	None	26.85 tok/s
RTX 5060 Ti 16GB	Blackwell	sm_120	448 GB/s	Q3_K_M-MTP	MTP $n = 2$	46.94 tok/s
RTX 5060 Ti 16GB	Blackwell	sm_120	448 GB/s	IQ4_XS-pure-MTP	MTP $n = 2$ + turbo3	54.31 tok/s
RTX 5060 Ti 16GB	Blackwell	sm_120	448 GB/s	IQ4_XS-pure-MTP	MTP $n = 2$ + kvarn4 (@32K)	38.10 tok/s

Turing Hardware Bottleneck Analysis

The Tesla T4 (sm_75) lacks native hardware acceleration for BF16 tensor operations. The 48 Gated DeltaNet recurrent scan layers execute sequentially in FP32, preventing batched speculative verification from scaling effectively. As a result, T4 throughput is physically bounded to ~25–33 tok/s regardless of quantization level.

6. Context Scaling and Memory Ceilings on 16GB VRAM

Memory utilization models on a 16,384 MiB physical VRAM budget with kvarn4 KV cache quantization:

Model Quantization	Model Weight VRAM	Draft Context VRAM	Maximum Usable Context (100% GPU)	Decode Speed at Max Context	Operating Status
IQ4_XS-pure-MTP	13.56 GiB	486 MiB	~48,000 tokens	~37.4 tok/s (5060 Ti)	Primary recommended profile
Atomic AD-IQ3_S	12.09 GiB	486 MiB	~72,000 tokens	~36.0 tok/s (5060 Ti)	High-context profile
IQ3_XXS-MTP	11.17 GiB	486 MiB	~88,000 tokens	35.19 tok/s (5060 Ti)	Extreme context; high quality degradation
IQ4_XS-pure (No MTP)	13.56 GiB	0 MiB	~80,000 tokens	18.64 tok/s (5060 Ti)	Stable non-speculative baseline
IQ4_XS-pure (FFN Offload -ot)	8.20 GiB GPU	0 MiB	128,000+ tokens	8.55 tok/s (5060 Ti)	CPU offloaded; memory safe
IQ4_XS-pure (Unified Memory)	Overflow to RAM	0 MiB	64,000+ tokens	1.45 tok/s	Fallback only; unusable for real-time

7. Inference Engine and Fork Comparison

Engine / Branch	Repository	Core Features	Build Support	Stability / Compatibility Notes
beellama.cpp	Anbeeld/beellama.cpp	kvarn2–kvarn8, precision tail, MTP, dflash	Source build required	Selected Engine. Native support for KVarN compression algorithms.
llama-cpp-turboquant	TheTom/llama-cpp-turboquant	turbo2–turbo4, FA all quants	Source build required	Slightly faster base decode on TurboQuant, lacks KVarN implementations.
buun-llama-cpp	spiritbuun/buun-llama-cpp	TCQ quant types, VBR, dflash optimization	Source build required	Validated alternative for non-standard quant formats.
Stock llama.cpp b10437	Official Release	Upstream standard	Prebuilt	Critical Bug: Segfaults in ggml_vec_dot_q3_K_q8_K on Qwen hybrid models.
Stock llama.cpp b10470+	Upstream Master	Upstream standard	Prebuilt / Source	Regression resolved; lacks kvarn and turbo KV cache types.
ExLlamaV3	Official Repository	Single-stream kernel acceleration	Prebuilt / Wheels	Incompatible with Turing (sm_75) and early hybrid attention kernels.
vLLM	Official Repository	Continuous batching, PagedAttention	Wheels / Docker	High VRAM allocation baseline; unsuited for single-stream 16GB deployments.

8. Build Instructions and Runtime Configurations

8.1 Compiling beellama.cpp

Build with CUDA support matching your target GPU architecture. The relevant NVIDIA RTX mappings are 75 / sm_75 for RTX 20-series (Turing), 86 / sm_86 for RTX 30-series (Ampere), 89 / sm_89 for RTX 40-series (Ada), and 120 / sm_120 for RTX 50-series (Blackwell). These are CUDA compute-capability targets used to compile the kernels, not benchmark results. Blackwell support requires CUDA 12.8 or newer.

git clone https://github.com/Anbeeld/beellama.cpp.git cd beellama.cpp cmake -B build \ -DGGML_CUDA=ON \ -DCMAKE_CUDA_ARCHITECTURES="75;86;89;120" \ -DCMAKE_BUILD_TYPE=Release \ -DGGML_CUDA_FA_ALL_QUANTS=ON cmake --build build -j$(nproc) --target llama-server llama-cli llama-perplexity 

8.2 Model Acquisition

• Balanced Model: unsloth/Qwen3.8-27B-MTP-GGUF -> Qwen3.8-27B-IQ4_XS-pure-MTP.gguf
• High-Context Model: AtomicChat/Qwen3.8-27B-GGUF -> Qwen3.8-27B-AD-IQ3_S-IQ3_XXS.gguf
• dflash Draft Model: ggml-org/Qwen3.6-27B-GGUF -> dflash-Qwen3.6-27B-Q8_0.gguf (Optionally requantize to Q4_0 via llama-quantize --allow-requantize)

8.3 CLI Execution Flags Explained

• -ngl 999: Offloads all 64 model layers and prediction heads to the GPU.
• --cache-type-k kvarn4 --cache-type-v kvarn4: Sets 4-bit variance-normalized quantization for both Key and Value caches.
• --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.0: Configures multi-token speculative decoding to evaluate 2 candidate tokens per step with greedy acceptance.
• -fa on: Enables FlashAttention (mandatory for KVarN acceleration kernels).
• -np 1: Forces single-slot execution. Default parallel slot allocations replicate recurrent state tensors across memory, causing severe throughput regressions on hybrid architectures.
• --fit off: Disables automatic context/layer shedding when memory allocation approaches physical VRAM limits.
• -t 8: Allocates 8 CPU worker threads for host orchestration.

9. Troubleshooting and Known Pitfalls

Issue	Root Cause	Resolution
Segmentation Fault at Load	Stock release b10437 bug in ggml_vec_dot_q3_K_q8_K	Deploy builds $ge text{b10470}$ or use beellama.cpp.
Severe Prefill Slowdown ($<10text{ tok/s}$)	Partial layer offload (-ngl 30) disables fused chunked Gated DeltaNet CUDA kernels	Maintain full GPU offload (-ngl 999) or execute fully on CPU (-ngl 0).
Throughput Regresses by ~10×	Default execution assigns n_parallel > 1	Append -np 1 to force single-slot memory layout.
Early Context OOM Crash	Server fit algorithm sheds layers dynamically near VRAM limit	Add --fit off to enforce configured allocation parameters.
MTP Initialization Failure	Attempting speculative execution on base non-MTP GGUFs	Use GGUFs containing bundled nextn draft heads.
glibc Symbol Errors on Prebuilts	Binaries built on Ubuntu 24.04 (glibc 2.38) executed on Debian 12 (glibc 2.36)	Compile directly from source on the target host environment.
Process Termination via pkill	Substring pattern matches active SSH command line arguments	Target exact process names: pgrep -x llama-server / pgrep -x llama-cli.