Qwen3.8-Flash-Next at 170K context on a single 96 GB card. ~110 tok/s.

I used the quantized n-gram to INT4, it's 32 GB, memory-mapped from disk.
I confirmed that it works great on 150-160k context, and i was watching all the time my VRAM usage while doing single thread long horizon things - the available VRAM should be enough to push it to over 170k and above)
The quality is there guys... It really is. It made a few complex html games and it figured out ways to play them itself without a browser (my ubuntu machine does not have any gui) and it kept improving and improving.... Here we go:

hf download primitive-ai/Qwen3.8-Flash-Next-NVFP4 \ --exclude "ple-bf16-*" --local-dir ./flash-next cd flash-next hf download primitive-ai/Qwen3.8-Flash-Next-PLE-quant \ --include "ples_int4/*" --local-dir . hf download primitive-ai/Qwen3.8-Flash-Next-PLE-quant \ worker_image_quant.py ple_layer_quant.py --local-dir . 

Skipping ple-bf16-* (saves 100 GB but breaks the index. So we trim that index):

import json p='model.safetensors.index.json'; d=json.load(open(p)); wm=d['weight_map'] drop=[k for k,v in wm.items() if v.startswith('ple-bf16-')] assert len(drop)==128 and all('ngram_embedding' in k for k in drop) for k in drop: del wm[k] json.dump(d, open(p,'w')) 

My intent was to fit the n-grams in my 64Gb of RAM, but at the end, n-grams and experts + kvcache all live inside the GPU's VRAM and its FAST!
76–125 tok/s single stream. The spread is MTP acceptance: ~87% on code and JSON, ~40% on just talking. Prefix caching hits 90%+ on a long horizon task. ~89 GB VRAM, ~33 GB page cache, 165-170K context, one GPU. Here's my full k0s yaml file (single server with a single Pro 6000).
I'm running it on my single node k0s and here is my yaml (cuda 13/580, ubuntu 24.04 no gui):

apiVersion: apps/v1 kind: Deployment metadata: name: vllm-qwen38-flash-next namespace: default spec: replicas: 1 strategy: type: Recreate # never two of these on one GPU selector: matchLabels: app: vllm-qwen38-flash-next template: metadata: labels: app: vllm-qwen38-flash-next spec: runtimeClassName: nvidia nodeSelector: nvidia.com/gpu.present: "true" tolerations: - effect: NoSchedule key: nvidia.com/gpu operator: Exists initContainers: - name: init-echo image: busybox:1.36 command: ["/bin/sh", "-c"] args: ['echo "I am here" > /opt/reservation/echo.txt'] volumeMounts: - mountPath: /opt/reservation name: reservation-volume containers: - name: vllm-server image: vllm/vllm-openai:qwen38-flash-next imagePullPolicy: IfNotPresent args: - --model - /model # ---- load-bearing for single-GPU PLE offload ---- - --distributed-executor-backend - mp # ------------------------------------------------- - --dtype - auto - --kv-cache-dtype - auto - --gpu-memory-utilization - "0.95" - --max-model-len - "173400" - --tensor-parallel-size - "1" - --pipeline-parallel-size - "1" - --limit-mm-per-prompt - '{"image":12,"video":2}' - --max-num-batched-tokens - "16384" - --max-num-seqs - "4" - --enable-chunked-prefill - --enable-prefix-caching - --no-enable-flashinfer-autotune - --speculative-config - '{"method":"mtp","num_speculative_tokens":3}' - --override-generation-config - '{"temperature":1,"top_p":0.95,"top_k":20}' - --enable-auto-tool-choice - --reasoning-parser - qwen3 - --tool-call-parser - qwen3_coder - --trust-remote-code - --api-key - key1 - --host - 0.0.0.0 - --port - "8990" - --served-model-name - qwen38-flash env: - name: VLLM_PLE_CPU_OFFLOAD value: "1" - name: VLLM_PLE_OFFLOAD_READY_TIMEOUT value: "1800" # Confirmed: worker_image_quant.py:419 reads this. Points at the # INT4 table dir; the overlay memory-maps it (MADV_RANDOM, mode "c"). - name: VLLM_PLE_QUANT_DIR value: /model/ples_int4 # Deliberately NOT setting VLLM_PLE_DISK_OFFLOAD_DIR (line 450) -- # that selects the BF16-table-on-NVMe path instead. - name: VLLM_LOGGING_LEVEL value: INFO - name: OMP_NUM_THREADS value: "1" - name: PYTORCH_CUDA_ALLOC_CONF value: max_split_size_mb:512 ports: - containerPort: 8990 protocol: TCP resources: limits: cpu: "12" nvidia.com/gpu: "1" requests: cpu: "8" nvidia.com/gpu: "1" securityContext: capabilities: add: ["IPC_LOCK", "SYS_ADMIN"] startupProbe: httpGet: path: /health port: 8990 periodSeconds: 15 failureThreshold: 80 # ~20 min; first boot loads the table readinessProbe: httpGet: path: /health port: 8990 periodSeconds: 20 failureThreshold: 3 lifecycle: preStop: exec: command: ["/bin/sh", "-c", "rm -f /opt/reservation/echo.txt"] volumeMounts: - mountPath: /model name: model-volume readOnly: true # --- two-file quantized-PLE overlay --- - mountPath: /usr/local/lib/python3.12/dist-packages/vllm/v1/ple_offload/worker.py name: ple-worker-overlay readOnly: true - mountPath: /usr/local/lib/python3.12/dist-packages/vllm/models/qwen3_8_flash_next/nvidia/ple_layer.py name: ple-layer-overlay readOnly: true - mountPath: /ples_int4 name: ple-tables readOnly: true # -------------------------------------- - mountPath: /dev/shm name: dshm - mountPath: /root/.cache/vllm name: vllm-cache - mountPath: /root/.triton name: triton-cache - mountPath: /opt/reservation name: reservation-volume volumes: - name: model-volume hostPath: path: /directory/models/Qwen3.8-Flash-Next-NVFP4 type: Directory - name: ple-worker-overlay hostPath: path: /directory/ple-overlay/worker_image_quant.py type: File - name: ple-tables hostPath: path: /directory/models/Qwen3.8-Flash-Next-NVFP4/ples_int4 type: Directory - name: ple-layer-overlay hostPath: path: /directory/ple-overlay/ple_layer_quant.py type: File - name: dshm emptyDir: medium: Memory sizeLimit: 32Gi - name: reservation-volume hostPath: path: /opt/reservation type: DirectoryOrCreate - name: vllm-cache hostPath: path: /var/cache/vllm type: DirectoryOrCreate - name: triton-cache hostPath: path: /var/cache/triton type: DirectoryOrCreate --- apiVersion: v1 kind: Service metadata: name: vllm-qwen38-flash-next namespace: default spec: type: NodePort selector: app: vllm-qwen38-flash-next ports: - name: http port: 8990 targetPort: 8990 nodePort: 32001 protocol: TCP 

Big Thank you to primitive-ai, whoever he is.