RAM Offloading with vLLM - tcclaviger appreciation post

Thanks to tcclaviger, vLLM now has expert RAM offloading support (link). This makes frontier models much more accessible on a local setup!

I was able to run the original DeepSeek-V4-Flash-Vision-Exp on four R9700s.

podman run --rm -it \ --init \ --network host \ --ulimit memlock=-1:-1 \ -v /models:/models:ro \ -v ~/.vllm-cache:/cache \ -e VLLM_ROCM_USE_AITER=0 \ -e ROCR_VISIBLE_DEVICES=0,1,2,3 \ -e VLLM_CACHE_ROOT=/cache/vllm \ -e TORCHINDUCTOR_CACHE_DIR=/cache/inductor \ -e TRITON_CACHE_DIR=/cache/triton \ --device /dev/kfd \ --device /dev/dri \ --group-add keep-groups \ --annotation run.oci.keep_original_groups=1 \ --security-opt label=disable \ --security-opt seccomp=unconfined \ --shm-size 160g \ docker.io/tcclaviger/vllm@sha256:ef99b3d07c3f15e7978528c7510762ba024df9ab4242070d8ed092cd4cc1a694 \ /models/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp \ --served-model-name DeepSeek-V4-Flash-Vision-Exp \ --tensor-parallel-size 4 \ --enable-expert-offload \ --expert-offload-mem 160 \ --reasoning-parser deepseek_v4 \ --reasoning-config '{"reasoning_parser":"deepseek_v4","reasoning_start_str":"","reasoning_end_str":""}' \ --tool-call-parser deepseek_v4 \ --enable-auto-tool-choice \ --max-num-seqs 8 \ --enable-prefix-caching \ --enable-chunked-prefill \ --max-num-batched-tokens 2048 \ --kv-cache-dtype fp8 \ --block-size 256 \ --max-model-len 256000 \ --gpu-memory-utilization 0.97 \ --mm-processor-cache-gb 4.0 \ --override-generation-config '{"max_tokens": 128000, "temperature": 1.0, "top_p": 0.95}' \ --speculative-config '{"method":"dspark","model":"/models/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp","num_speculative_tokens":3,"draft_sample_method":"probabilistic","enable_adaptive_verification":false}' \ --compilation-config '{"cudagraph_capture_sizes": [4,8,12,16], "max_cudagraph_capture_size": 16}' \ --host 0.0.0.0 \ --port 8090