News / #inference Tag Inference 500 articles archived under #inference · RSS Sign in to follow r/LocalLLaMA community 18h ago Discover AI: A Learning Guide - 51 hands-on labs for learning how AI actually works. This is an offline app with 51 hands-on labs for learning how AI actually works, from neurons to agents and more... **A bit of insight** - 51 labs in 6 groups, from the basics (what a neuron is, gradient descent) through attention, RAG, agents, fine-tuning, quantization, serving… 32 r/LocalLLaMA community 22h ago Someone is running Qwen 27B at ~20 tok/s on a jailbroken PS5's GPU   submitted by   /u/LatentSpacer [link]   [comments] 33 r/LocalLLaMA community 1d ago SonicBoom: 52 tok/s decode on a laptop RTX 3050 (llama.cpp: 45) with the same quantized kernels, the difference is a graph compiler. No Python. I’ve been building SonicBoom, a native tensor runtime written in C++/CUDA with a C ABI and no Python anywhere in the stack. I just tagged v0.0.1, and the first target is local LLM inference on consumer GPUs. Numbers (Gemma 4 E4B-it Q3_K_M on a laptop RTX 3050 8 GB, both runtimes… 12 llama.cpp releases dev-tools 1d ago b11577 model: fix vllm d3 multi seq case ( #30408 ) model: fix vllm d3 multi seq case add TODO Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/55219202 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled)… 22 llama.cpp releases dev-tools 2d ago b11571 model: support vllm decision 2 model ( #30378 ) models: support pplx-decider vllm d2 update clarify naming rule stricter validation fix ci rm redundant filter_tensors Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/55070168 macOS/iOS:… 10 r/LocalLLaMA community 2d ago graphsignal-top: a `top` for inference GPUs in the open-source Graphsignal profiler, kernels ranked by time, live, with readable names A few weeks ago we posted the open-source Graphsignal profiler here: wrap vllm serve / sglang serve / whatever in graphsignal-run and it serves everything it measures as JSON, mainly so an agent can read it in a loop. Turns out we kept opening the JSON ourselves to see what the… 20 llama.cpp releases dev-tools 2d ago b11572: model: support vllm d3 models (#30383) models: support pplx-decider vllm d2 update clarify naming rule stricter validation d3 28 r/LocalLLaMA community 2d ago A "free" multi-modal "embedding" model (already) included with your LLM: vLLM plugin + upstream PR Disclaimer: I work for NuMind, the company behind this plugin. TLDR: Text/image (simili-)embeddings from the LLM you already have loaded :-) I've been working on making it easier to obtain embeddings from a LLM (preferably multi-modal) that is already being served by vLLM. The… 37 r/LocalLLaMA community 2d ago Qwen3.8-Flash-Next on six 3090s with vLLM: 7 users at 262k, 6.5k t/s prefill, 57 t/s decode solo / 206 t/s aggregate with all 7 generating. Fixes for the upstream included. Write-up, serve command and raw data: https://github.com/lukolszewski/vllm This shows vLLM fixes and setup that let you put PLE on SSD and save 108GB of system RAM, use int8 kv and squeeze more users in your VRAM without the performace hit of fp8. Speeds in comparison to my… 34 r/LocalLLaMA community 2d ago Decision 3.0: open, multimodal decision models from 0.6B to 27B. Ranks #1 on both Jev Decision Index boards With the vLLM Semantic Router team we released Decision 3.0 (d3) , our new SOTA decision model family ranging from 0.6B to 27B, Apache-2.0, all reading text, images and video. How it works - Built on Qwen3.5 (0.8B / 2B / 4B / 9B) and Qwen3.8-27B. The parameter counts include the… 38 arXiv — Machine Learning research 2d ago Byzantine-Resilient Federated Large Language Model via Hypergraph Message Passing arXiv:2610.12711v1 Announce Type: new Abstract: Federated Large Language Models (FedLLMs) are promising to deliver privacy-preserving, collaborative intelligence. The decentralized training process is vulnerable to Byzantine attacks that corrupt semantic consistency and factual… 24 arXiv — Machine Learning research 2d ago Personalized Federated Direct Preference Optimization via LoRA Adapters arXiv:2610.12885v1 Announce Type: new Abstract: Federated Direct Preference Optimization (FedDPO) enables privacy-preserving alignment of language models across distributed agents without centralizing sensitive preference data. However, existing FedDPO analysis assumes all… 24 arXiv — Machine Learning research 2d ago Continually Post-training Small Models via Synthetic Self-distillation arXiv:2610.13113v1 Announce Type: new Abstract: Continual post-training faces a dual challenge of acquisition and retention: models must learn new capabilities while preserving prior knowledge. While Self-Distillation Fine-Tuning (SDFT) mitigates this using self-supervision with… 25 arXiv — Machine Learning research 2d ago DuoRate: Two-Rate Per-Expert Precision Allocation for Mixture-of-Experts Serving arXiv:2610.13328v1 Announce Type: new Abstract: Serving a Mixture-of-Experts (MoE) model pays for expert precision twice: the resident footprint pays for every expert the router can reach, while each batch-one decode step pays only for the experts it selects. These are two… 38 arXiv — NLP / Computation & Language research 2d ago Self-Preservation in Language Models arXiv:2610.12602v1 Announce Type: new Abstract: Continual learning (CL) seeks to adapt pretrained models while preserving prior capabilities. Yet the tasks supplied for preservation cover only part of what a model knows, while forgetting can extend beyond them. Recent work shows… 14 arXiv — NLP / Computation & Language research 2d ago perturber: A Stateless Service for Semantics-Preserving Text Perturbations arXiv:2610.12667v1 Announce Type: new Abstract: Large language model (LLM) evaluations on clean benchmarks overstate model robustness: real inputs arrive perturbed, and these meaning-preserving variants routinely move model answers. Studying this systematically requires applying… 21 arXiv — NLP / Computation & Language research 2d ago CoBe: A Benchmark For Conversational Counterfactual Text Editing arXiv:2610.13057v1 Announce Type: new Abstract: Counterfactual text editing is the ability to revise a given text in light of a hypothetical change while preserving facts not causally affected by that change. When asked what would have happened if a doctor had not recommended… 32 arXiv — NLP / Computation & Language research 2d ago KV-Repurpose: A Testbed for Evaluating Non-Prefix KV-Cache Reuse arXiv:2610.13060v1 Announce Type: new Abstract: KV-cache reuse in large language model serving is commonly limited to exact prompt prefixes, preventing reuse when the same retrieved document appears after different system prompts, conversation histories, or dynamic metadata. We… 6 arXiv — NLP / Computation & Language research 2d ago Taking Tokenization off the Host for Agentic LLM Serving arXiv:2610.13325v1 Announce Type: cross Abstract: Agentic workloads resend long contexts that barely change between calls. Prefix caching now skips most of their prefill, yet every request must still be tokenized in full before the cache can even be consulted. Tokenization, a… 5 r/LocalLLaMA community 3d ago "Strata" for GLM5.3 Flash is here for some! Project Maya I stumbled across this as I was currently having glm5.3 flash run only around 10tok/s basically unusable. With Maya i am running 30 tok/s now. GLM5.3 feels even with this low quant like a much more enjoyable model than qwen 3.8 flash next so far. Still testing . This is very… 29 r/LocalLLaMA community 3d ago vllm-ascend updates (GLM5.3-flash) on dual 310p Ascend cards Alright, I learend a few valuable lessons since I posted about a week ago, and I've made more strides with the DDR4x Ascend 96GB 310P cards I purchased, so here we go, meet the new friendly and less verbose me. GLM-5.3-Flash is a 320-billion-parameter MoE model, with about 18… 8 r/MachineLearning community 3d ago Is a second MSc worth it? [D] I’m an AI research engineer based in Africa with about 3 YoEs in RL, LLMs, post-training, and systems efficiency (CUDA/vLLM). However, on a smaller scale because of compute limits. My undergrad and Master’s are both in communications engineering. I did a 5th-year Master's at my… 6 r/LocalLLaMA community 3d ago UPDATE: Qwen 3.8 27B 140 tok/s on single RTX 3090 Megakernel: KL divergence 0.0009 vs llama.cpp This is a follow-up to my post from yesterday ( https://www.reddit.com/r/LocalLLaMA/comments/1x2erdj/qwen3827b_on_a_single_3090_140_toks_on_code_with/ ) detailing the CUDA megakernel. Thank you all for the positive feedback and PRs! RECAP: The megakernel is a CUDA engine for… 12 r/LocalLLaMA community 3d ago Now you can grow Bonsai on your potato No more excuses for GPU-poor folks not to start LLMing! Full 27B-class reasoning in ternary transformer weights, for llama.cpp (CUDA, Metal, CPU) ~9.3x smaller than FP16 (ideal) | 98.2% of FP16 intelligence retained | ~47 tok/s on an Apple M5 Max laptop Highlights ~5.9 GB… 6 r/LocalLLaMA community 3d ago M5 Ultra 256GB or 2x DGX Spark? Which one? Currently contemplating adding either M5 Ultra 256GB or dual DGX Spark in addition to existing 5090. I have been researching for past few days and have zeroed in on dual sparks because higher throughput for agentic coding / multiple parallel requests and prompt processing… 34 r/LocalLLaMA community 3d ago Running Next Flash IQ3_XXS at ~70 tok/s with 100k context or 2 instances of Qwen 3.6 35B A3B IQ4 at ~145 tok/s with 256k all on $500 of ex mining BC-250 boards This will be my third update on the bc-250 cluster and for my first forray into local ai I have been having a blast. This has turned from a weekend experiment into something I think I will use in the future. I am running 4 BC-250 boards connected by 2.5gb ethernet adapters to a… 13 r/LocalLLaMA community 4d ago Qwen3.8-27B on a single 3090: 140 tok/s on code with a custom megakernel I've been using Claude Opus 5.5 to speed up Qwen3.8-27B on my PC (rtx 3090), it wrote a CUDA megakernel that is 1.4-1.9x faster than llama.cpp depending on the task/context length. The results and code are below: Results (same hardware, my megakernel vs llama.cpp with MTP): -… 16 r/LocalLLaMA community 4d ago Qwen 3.6 35B A3B: 131K context + vision on 6GB VRAM Running HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive Q4_K_M with llama.cpp at ~600 tok/s prefill and 23 tok/s decode, 131k context window, Q8 KV cache - on an RTX 2060 6GB + 32GB DDR4 RAM. Speeds start at ~600 tok/s prefill / 23 tok/s decode on an empty KV cache. As… 21 r/LocalLLaMA community 4d ago Basalt: Flash-Next at 665 tok/s structured, 354 prose on a 5090 + 5060 Ti (2.6x Strata) Basalt: Blackwell inference engine Hi all! So over the last week I've been working on a Strata fork that's heavily tuned and can achieve throughput up to 2.6x what Strata usually does on the same weights. It's designed as a specialized engine that only supports Qwen3.8… 7 r/LocalLLaMA community 4d ago PR merged: `/v1/systemone` support in vllm   submitted by   /u/413205 [link]   [comments] 21 r/LocalLLaMA community 4d ago Qwen Flash Next @ 137 tok/s & 3,497 tok/s Prefill w/ 512k context on a 5090, 192gb ram, Windows Build, comparing Strata and Infernix Strata vs Infernix on the same model, RTX 5090 — A/B at 262k and 512k context Hi folks, another inference engine post for your feed. I was blown away by Strata, but saw someone else post about Infernix with some wild claims, and it doesn't seem so well known, so I've spent a day… 19 r/LocalLLaMA community 4d ago Qwen3.8-27B UD-IQ4_XS Heretic + MTP on a 16 GB card with 55-68 tok/s (24gb and 12gb versions available too) I wanted the uncensored Qwen3.8-27B (llmfan46's Heretic build, MTP head preserved) on my RTX 4080 with MTP on (because I've been testing TONS of configs, and realized that would be the one) And none of the published quants were built for that: the good IQ4_XS doesn't leave room… 4 r/LocalLLaMA community 5d ago Qwen 3.8 Flash Next-GSQ-RCO-IQ2_XS at ~21 tok/s on just an RTX 3060 12GB + 16GB DDR4 RAM(No gate pruning, 100% bit-exact) (Don't judge by the screenshot, the cache is cold. It hits 24+ tok/s with a warm cache!) About two months ago, I made a post here asking whether predicting which MoE experts would be used on the next token could actually help speed up CPU/GPU offloading. Original post: Tried… 11 r/LocalLLaMA community 5d ago Qwen3.8-Flash-Next-GSQ-RCO (IQ3_S): ~20-30 tok/sec decode & 300-90k tok/sec prefill on 12GB VRAM + 32GB RAM + NVME I got Qwen3.8-Flash-Next-GSQ-RCO-Abliterated running at IQ3_S with just 12GB VRAM and 32GB system RAM, achieving 20-30 tok/sec decode (Q2 achieves 39-45 tok/s) & 300 to ~90 thousand tok/sec prefill @ 131k context, on a custom fork of Strata. This fork has tonnes of architectural… 4 OpenAI official-blog 5d ago Sophos cuts threat investigation time by 96% with OpenAI Daybreak Discover how Sophos uses OpenAI’s Daybreak to cut cyber-threat investigation time by 96% and automate 52% of MDR cases while preserving human oversight. 10 r/LocalLLaMA community 5d ago Running the uncensored Qwen3.8-27B (HauhauCS) on a 4090 at 262K context and ~130 tok/s HauhauCS ships their uncensored Qwen3.8-27B as GGUF only. NInfer, a C++/CUDA wanted its own format. Now the same model that ran at 91.6 tok/s / 131K under llama.cpp does: 262K context (the model's full native window) ~130 tok/s decode with MTP3, 70.8% acceptance 3,591 tok/s… 5 arXiv — Machine Learning research 5d ago Stability-Plasticity Balance via Singular-Vector Selection in LLM Continual Learning arXiv:2610.11076v1 Announce Type: new Abstract: Domain-specific continual adaptation of LLMs risks catastrophic forgetting, creating a fundamental tension between acquiring new capabilities and preserving those learned during pretraining. PEFT mitigates this problem by… 31 arXiv — Machine Learning research 5d ago Bernoulli Flow Models: Self-Consistent Generative Modeling for Binary Data arXiv:2610.11362v1 Announce Type: new Abstract: Binary diffusion models typically require a large number of function evaluations (NFEs) to generate high-quality samples, making practical inference computationally expensive. Reducing NFEs while preserving sample quality without… 35 arXiv — NLP / Computation & Language research 5d ago Diffu-LoRA: A Novel Low-Rank Adaptation for Personalized Diffusion Models arXiv:2610.10550v1 Announce Type: new Abstract: Personalizing text-to-image diffusion models from a few reference images requires preserving subject identity while following prompts that describe new contexts. Full-model fine-tuning is parameter-intensive, whereas low-rank… 22 arXiv — NLP / Computation & Language research 5d ago From Retrieval to Reconstruction: Constructing Evolvable Cognitive Memory for Long-Term Dialogue arXiv:2610.11314v1 Announce Type: new Abstract: Large Language Models (LLMs) serving as long-term dialogue agents require memory systems that support reliable reasoning over extended interactions. However, existing Retrieval-Augmented Generation (RAG) frameworks typically treat… 29 arXiv — NLP / Computation & Language research 5d ago Not Every Change Is Necessary: Recoverable Drift in Large Language Model Unlearning arXiv:2610.11915v1 Announce Type: new Abstract: Machine unlearning in large language models aims to remove unwanted knowledge while preserving the model's remaining capabilities. Although existing methods use retention objectives or restrict where edits occur, achieving the… 10 arXiv — NLP / Computation & Language research 5d ago TokenRouter: Efficient Serving System for Token-Level LLM Routing arXiv:2610.12242v1 Announce Type: new Abstract: Large language model (LLM) routing distributes inference work across different models, advancing the cost-quality Pareto frontier of LLM serving. While coarse-grained routing at the session or query level has been widely adopted in… 5 vLLM releases dev-tools 5d ago proto-v0.5.0 Release vllm-proto 0.5.0 14 r/LocalLLaMA community 5d ago Qwen3.8-27B: 159 tok/s on R9700, 64 tok/s on Strix Halo LemonSeed Studio is an iPad editor/IDE with on-device inference on an AMD GPU in a Thunderbolt enclosure. It embeds the unmodified upstream Linux amdgpu + amdkfd driver (mac_linuxgpu) as a PCIDriverKit extension, and runs LemonSeed Engine (LSE) on the GPU. In the photos: iPad… 36 r/LocalLLaMA community 6d ago $2800 rig with 8x Radeon Pro V620 (256 GB VRAM) + custom vLLM fork = Qwen3.8-Flash-Next at 60 to 100 t/s decode and 3000+ t/s prefill Post title is slightly misleading, I don't think you can get these for $350 each anymore but they're still pretty cheap all things considered. They're Radeon Pro V620's which are older RDNA2 enterprise cloud gaming cards with 32 GB VRAM. (Ignore the RTX 4090 on the side, it's… 37 r/LocalLLaMA community 6d ago Best current R9700 inference engine? There are way too many forks to keep track of, so I've gotten lost. As far as I can tell, Radiance VLLM is best for models that fit in GPUs while some form of llama.cpp is probably best for MOE RAM-spill? My specific current goal is to run GLM5.3-Flash over 6 R9700s + system RAM… 32 arXiv — Machine Learning research 6d ago HCPN-GCN: Scaling Hierarchical Prototype Networks with Cone Geometry for Continual Graph Learning arXiv:2610.08823v1 Announce Type: new Abstract: Continual Graph Learning (CGL) aims to incrementally learn from graph-structured data while preserving knowledge acquired from previous tasks. A major challenge in this setting is catastrophic forgetting, where learning new tasks… 10 arXiv — Machine Learning research 6d ago Removing Information Content Does Not Certify Tamper Resistance in Open-Weight Models arXiv:2610.09004v1 Announce Type: new Abstract: Does removing harmful information make open-weight models resistant to fine-tuning attacks? We show that mutual information at release alone cannot universally certify slow recovery. Function-preserving reparameterizations leave… 8 arXiv — Machine Learning research 6d ago Denoising Blocks, Not Tokens: Efficient Compressed Continuous Diffusion with Branching Token Realization arXiv:2610.09311v1 Announce Type: new Abstract: Diffusion language models (DLMs) generate text through iterative parallel refinement, offering the potential for higher throughput than autoregressive (AR) decoding. However, most DLMs still maintain one generative state per token,… 37 arXiv — Machine Learning research 6d ago Shared Low-rank Basis Factorization for Data-free Mixture-of-Experts Compression arXiv:2610.09342v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) large language models decouple capacity from compute through sparse routing, but their large parameter count creates storage and serving challenges. We analyze three MoE compression families: expert… 28 Page 1 of 10 · 500 articles Older →
