News / #inference Tag Inference 500 articles archived under #inference · RSS Sign in to follow r/LocalLLaMA community 1mo ago Solved: LLM inference on Windows was 2–3x slower when the server window wasn't focused The fix: run the server detached/headless instead of keeping it attached to a console window. RTX 5090, ~27B NVFP4 model via ninfer: Terminal focused: 130–200 tok/s Terminal unfocused: 50–60 tok/s Click the terminal → immediately back to 130+ tok/s At first I thought GPU… 17 r/LocalLLaMA community 1mo ago 1-bit 27B in the browser: 25–30 tok/s on a 6 GB RTX 3060 Laptop (WebGPU, no install) mentria.ai is a browser inference engine I've been building solo, from scratch in WebGPU/WGSL. This week it crossed a milestone I had been chasing for a while: a 27B one-bit model answering at up to 30 tokens/s on an RTX 3060 Laptop GPU with 6 GB of VRAM, in Chrome, from a web… 22 r/LocalLLaMA community 1mo ago DeepSeek-V4-Flash-Vision-Exp (285B MoE) on 10-12x RTX 3090 — spec decoding, vision Running the full deepseek-ai/DeepSeek-V4-Flash-Vision-Exp on consumer Ampere — 10-12x RTX 3090, SM86-compatible vLLM build. 285B MoE, FP4 experts + FP8 attention, 157 GB weights. Highlights: - **60+ tok/s** decode, DSpark spec (k=3) on 10 GPUs (TP2xPP5), at a 240 W cap - **120+… 17 Hugging Face Daily Papers research 1mo ago Reason Through the Latent! Making Latent Visual Reasoning Necessary Abstract CVRR enforces recurrent hidden-state computation as the required image-conditioned pathway for visual reasoning, preserving model competence while distinguishing latent information from actual predictive use. Generated by thinkingmachines/Inkling-Small Latent visual… 35 r/LocalLLaMA community 1mo ago Is RX 6800 + 6800 XT a sensible upgrade from 2x RTX 2060 OC 12GB for llama.cpp? I’m currently running llama.cpp on two RTX 2060 12GB cards, so 24GB total VRAM. With Qwen3.8 27B IQ4_XS at 131k context I’m getting around 45 tok/s, which is actually pretty good for this setup. I found a deal on an RX 6800 16GB and an RX 6800 XT 16GB, so I’d be going from 24GB… 8 r/LocalLLaMA community 1mo ago For Strix Halo - Official llama.cpp isn't ideal and how to highest possible throughput I've been making a lot of comments about optimal setup for Strix Halo (gfx1150) and from my observation, 90% of our community is using offcial llama.cpp for it, which is NOT optimized for Strix Halo at all, official llama.cpp is having extremely hard time to reach 50% hardware… 20 r/LocalLLaMA community 1mo ago Higher acceptance length, slower prose: Ling’s n=1/2/3 MTP test on one Spark The missing control is visible in sudoingX’s Ling-3.0-flash benchmark graphics. The earlier table leaves Ling’s no-speculation baseline as “not measured.” The later code/prose graphic fills it in: about 23 tok/s without the drafter, against 40.9 on code and 38.7 on prose with… 31 r/LocalLLaMA community 1mo ago Qwen 3.8 Next Flash is really really REALLY verbose.. Long time user of 3.6 27b, switched over to Next Flash since it's a logical step up even from 3.8 27b. It's soooo verbose, i'm talking 13 minutes of thinking time on single turn coding requests at approximately 150 tokens per second tg and 7000 tokens per second pp. It's… 24 arXiv — NLP / Computation & Language research 1mo ago When Load-Balancing Goes Too Far: Expert Pruning in Over-Dispersed Mixture-of-Experts Models arXiv:2609.04453v1 Announce Type: cross Abstract: Expert pruning reduces the memory and serving cost of Mixture-of-Experts (MoE) models by removing low-importance experts identified by the router, assuming router probabilities provide a reliable importance signal. We observe… 17 arXiv — Machine Learning research 1mo ago RegionFed: Federated Learning for Personalized Query Understanding in Heterogeneous Retail Environments arXiv:2609.05403v1 Announce Type: new Abstract: Retail search systems serve diverse geographic regions with distinct query patterns, vocabularies, and product preferences, creating significant data heterogeneity that challenges both privacy-preserving training and model… 22 arXiv — Machine Learning research 1mo ago Interface-Induced Trajectory Censoring arXiv:2609.03966v1 Announce Type: cross Abstract: Agent evaluations report a tool-call rate read off the serving stack. That number can be zero while the model is emitting well-formed calls: the interface censors the trajectory before anything downstream sees it. On BFCL v4's… 38 arXiv — NLP / Computation & Language research 1mo ago Scale-QLoRA: Code-Invariant Adapter Merging for Native 4-bit Microscaling LLMs arXiv:2609.04526v1 Announce Type: new Abstract: Merging a LoRA adapter into its base model is standard deployment practice: it removes the runtime adapter's per-forward overhead and leaves a single standalone checkpoint any serving stack can load. On a native 4-bit microscaling… 27 arXiv — NLP / Computation & Language research 1mo ago EuroAlpaca: Task-Preserving Localisation of Instruction Data for European Languages arXiv:2609.05043v1 Announce Type: new Abstract: Machine translation (MT) offers a scalable way to extend English instruction-tuning data to multiple languages, but it can distort task-critical constraints and required outputs, creating corrupted training examples and degrading… 17 r/LocalLLaMA community 1mo ago LayerStoRm open-source expert streaming: 1M context GLM-5.3-Flash [UD-Q4_K_XL] at 24.5 tok/s @8k on just 2× RTX 5090 + 2× RTX 5080 (186 GiB MoE on 96 GB VRAM) LayerStoRm: GLM-5.3-Flash UD-Q4_K_XL (186 GiB) at 1M context on 2× RTX 5090 + 2× RTX 5080 (96 GB VRAM total) using RAM for the pinned experts. LayerStoRm is a (still experimental) MIT-licensed continuous expert-streaming inference engine: it runs MoE models far larger than your… 33 r/LocalLLaMA community 1mo ago vllm + p2p driver hack + qwen 3.8 27B vs llamacpp + qwen flash next ? Hi everyone I'm running four rtx 4090, 64GB ram, on a threadripper pro motherboard so all PCIe x16 ports, as a homelab machine for coding. I was migrating from vllm + qwen 3.8 27B (fp8+256k kv cache) to llamacpp + qwen flash next iq4xs + 8 bit cache 200k kv cache... until… 34 r/LocalLLaMA community 1mo ago 2x R9700, 64 GB DDR5 is an absolute beast machine with vLLM Radiance / R9V and Qwen 3.8 27b and Flash next I've been tinkering with local LLMs since the beginning of the year when I had an Intel Arc B580 and 32 GB of DDR5. Curiosity got the best of me and I bought the first R9700 about half a year ago, also because I wanted to upgrade my gaming graphics for 4k. As the 5090 was about… 14 r/LocalLLaMA community 1mo ago Validate your local LLM advertised KV cache against real pressure; see exactly how old contexts get evicted from cache Hello, I'm a bit obsessed with cache management on local LLMs. For the last few days I've been working on cache management on vLLM with my 2x DGX Spark and DeepSeek v4 Flash 0731. I felt something was off so I investigated, found issues, fixed them, but needed a way to validate… 22 r/LocalLLaMA community 1mo ago Which qwen for vllm? Hugging face has over 300 versions of qwen3.8-27b. I have no idea how to identify the beat model to download and use. Should I sort by “most likes” or “most downloads”? Is there any sort of other guidance on which model to select? I’m finally getting my local linux box up and… 4 r/LocalLLaMA community 1mo ago Qwen3.8-Flash-Next-oQ4e-mtp: 45 tok/s on M4 Max, 25 tok/s on M2 Ultra for local inference — llm-bench.io Qwen 3.8 Flash Next gives similar speed than Qwen3.8 27B on Apple Silicon.   submitted by   /u/DerTomsn [link]   [comments] 26 r/LocalLLaMA community 1mo ago NInfer vs llama.cpp vs vLLM: quality + speed comparison for Qwen3.8-27B NVFP4 on RTX 5090 I've been running Qwen3.8-27B as a local inference server for a production content intelligence pipeline (HVAC industry stuff, lots of long-context retrieval and structured extraction). I have been watching other redditors post their custom configurations, and I wanted to share… 9 r/LocalLLaMA community 1mo ago How to estimate tokens/sec for your hardware We all want more tokens per second but I keep seeing confusion on what to expect for given hardware. For the decoding phase (TG/s) to produce one token all the model weights and KV cache needs to be read from VRAM. The compute isn't the bottleneck, only memory bandwidth. This… 23 r/LocalLLaMA community 1mo ago Updated my benchmark with a new vLLM based recipe for Qwen 3.8 Flash Next : now up to 98/100 (instead of 91 previously) I was using: weights https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4 with the optimized SGLANG (patched) from https://old.reddit.com/r/BlackwellPerformance/comments/1w04xb7/qwen38_flashnext_on_1x_rtx_pro_6000_171_ts_c1_428/ Now I'm using: weights (AWQ W4A16) from:… 11 r/LocalLLaMA community 1mo ago You can now run a 90M conversational LLM on the Sony PSP (hardware from 2004). Doesn't get more local than this. Github link: https://github.com/thatblend/LLMPSP I wanted to see what the PSP can theoretically handle and I got my answer - a 90M model is about the max it can do without atrocious inference speeds. It's running around 0.5 - 0.6 tokens per second, which is very slow, but it's… 6 Hugging Face Daily Papers research 1mo ago Let Confidence Change, Not the Prediction: Prediction-Preserving Repair for Post-hoc Calibration Abstract CORD is a post-fit adapter that repairs calibrated probability vectors to exactly preserve original top-1 predictions while maintaining calibration quality. Generated by thinkingmachines/Inkling-Small Post-hoc calibration corrects reported confidence, yet a multiclass… 14 arXiv — NLP / Computation & Language research 1mo ago Privacy-Preserving Heterogeneous Multi-LLM Federated Inference for Cognitive Diagnosis arXiv:2609.02947v1 Announce Type: cross Abstract: Significant challenges remain in AI-driven educational systems in balancing privacy preservation with accurate cognitive diagnosis. To overcome this, we propose a federated inference framework in which several commercial LLM APIs… 36 arXiv — Machine Learning research 1mo ago Privacy-Preserving Topology-Guided Safety for LLM-Based Multi-Agent Systems via Federated Graph Learning arXiv:2609.02967v1 Announce Type: cross Abstract: Topology-guided safeguards for LLM-based multi-agent systems (MAS) train a GNN over the inter-agent communication graph to localize risky agents and intervene on the topology---but they assume one operator can pool all labeled… 11 arXiv — Machine Learning research 1mo ago Privacy Leakage in Federated Learning: Gradient-Based Client Identity Inference and Defenses for Inertial Sensing in Vehicular Edge Networks arXiv:2609.02971v1 Announce Type: cross Abstract: As vehicular networks move toward 5G/6G edge intelligence, federated learning (FL) is widely promoted as a privacy-preserving way for vehicles and infrastructure to train shared models without exposing raw sensor data. Yet the… 22 arXiv — NLP / Computation & Language research 1mo ago Lost in Reordering: Structural Sensitivity of Multilingual LLMs under Semantics-Preserving Perturbations arXiv:2609.03511v1 Announce Type: new Abstract: Large Language Models (LLMs) demonstrate strong multilingual reasoning performance, yet their robustness to semantics-preserving structural variation remains underexplored, particularly for relatively free word-order languages. We… 4 r/LocalLLaMA community 1mo ago Qwen 3.8 27B Vs. Qwen 3.6 27B on oMLX Quality: 81.1 → 87.7 (+8%) Speed: 35 → 29 tok/s (−16%) Runtime: 8m51s → 44m39s (5x longer) Output tokens: 18K → 78K (🤯) Noticeably better quality, but you're paying for it with tokens and time. Full benchmark results (all hardware, all quants): llm-bench.io Qwen3.6-27B Vs.… 28 r/LocalLLaMA community 1mo ago Qwen3.8-Flash-Next MTP merged in ik_llama.cpp (integrated head or separate -md file)... 45 → 90 tok/s on a 5090 + 128GB, works down to a 12GB 4070 ik_llama.cpp merged qwen4exp MTP support yesterday (PR #2369, mine, reviewed and tested by four other people on their own hardware). It's on main now, no fork or patch needed. Posting since the last couple threads had people saying MTP for this model only exists as an unsloth… 34 r/LocalLLaMA community 1mo ago My RULE of Thumb of choosing a models This is mostly for setting up for expectation, since personally without LLM i could take 3 days (15 hours of active programming) to debug or implement a feature, but with Qwen 27B (even before Qwen 3.8) it take 4 hours. And yes 0.5 tok/s is human, not accounting of deletion and… 37 arXiv — Machine Learning research 1mo ago OutageDiT: A Generative Foundation Model for Power Outage Forecasting and Scenario Simulation arXiv:2609.01896v1 Announce Type: new Abstract: Power-outage planning requires scenarios before an event occurs. These scenarios must represent uncertainty in magnitude, timing, and duration while preserving temporal dependence. However, severe events are rare, and data from any… 14 arXiv — Machine Learning research 1mo ago Source-Free Class Relearning: Diagnosing Forgetting in Class Unlearning arXiv:2609.02018v1 Announce Type: new Abstract: Class unlearning aims to remove a model's ability to recognize designated forget classes while preserving performance on retain classes. However, low forget accuracy after unlearning does not necessarily mean the class structure… 16 arXiv — NLP / Computation & Language research 1mo ago XMerge: Cross-Axis Selection and Reconstructive Layer Merging for LLM Depth Compression arXiv:2609.02083v1 Announce Type: cross Abstract: Removing complete transformer layers preserves a standard serving architecture, but existing depth-compression methods can lose substantial quality, and the loss varies unpredictably across models. We introduce XMerge, a… 10 arXiv — Machine Learning research 1mo ago Exact Limits of Random Projections for Preserving Geometry: Distance Recovery, Nearest-Neighbor Rankings, and Covariance Shape in Gaussian Models arXiv:2609.02155v1 Announce Type: new Abstract: The Johnson-Lindenstrauss (JL) lemma guarantees that a random projection of $n$ points to $m=O(\varepsilon^{-2}\log n)$ dimensions preserves pairwise squared distances within relative error $\varepsilon$ with high probability, and… 16 arXiv — Machine Learning research 1mo ago Similarity-Aware Personalized Federated Learning in Heterogeneous Environments arXiv:2609.02241v1 Announce Type: new Abstract: Federated Learning (FL) allows decentralized clients to train models collaboratively while preserving data privacy. However, distribution mismatch across clients often leads to poor global generalization and degraded local… 33 arXiv — Machine Learning research 1mo ago Online Reinforcement Learning in the Met Office Unified Model through Distributed Model-Agent Coupling arXiv:2609.02566v1 Announce Type: new Abstract: Machine-learnt corrections can complement numerical weather prediction only if they adapt to the evolving model state while preserving dynamical consistency and numerical stability. To test this within a global forecasting model,… 24 arXiv — NLP / Computation & Language research 1mo ago HyperStyler: Low-resource Authorship Style Transfer via Context-aware Style Navigation and Hypernetworks arXiv:2609.02772v1 Announce Type: new Abstract: Low-resource authorship style transfer (LAST) aims to rewrite text into the style of an arbitrary target author using only a few reference examples while preserving the original meaning. Existing methods often struggle to achieve… 33 Hugging Face Daily Papers research 1mo ago Influence-Directed Distillation: Solving the Diversity Bottleneck in Sampled-Token On-Policy Distillation Abstract Influence-Directed Adaptive On-Policy Distillation improves diversity transfer in reasoning model distillation by selectively preserving entropy-expanding updates and replacing entropy-contracting ones with adaptive advantage shrinkage. Generated by… 23 r/LocalLLaMA community 1mo ago Quad R9700 AI Pro with vLLM-Radiance easily reaching 17,6k PP https://preview.redd.it/74bmvel9b5nh1.png?width=1602&format=png&auto=webp&s=0d0c1adaa016a486ffd97c4c466e980dc611b139 I've only recently started looking deeper into vLLM after running llama.cpp for a good while. Initially vLLM (official repo) was terribly slow on my four R9700s… 10 Hugging Face Daily Papers research 1mo ago From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix Abstract A smaller self-hosted LLM trained with separate GRPO experts merged via SLERP outperforms a much larger baseline on instruction following, function-calling, and internal tasks while serving half of platform traffic at lower cost. Generated by… 31 arXiv — Machine Learning research 1mo ago QTEA: Ternary LLMs with Sparse Residual Salient Weight and By-Column Optimization arXiv:2609.00224v1 Announce Type: new Abstract: Weight-only post-training quantization (PTQ) can alleviate the computational burden of serving large language models (LLMs) at scale. However, existing PTQ methods often fail to generalize across models and suffer severe accuracy… 36 arXiv — Machine Learning research 1mo ago Let Confidence Change, Not the Prediction: Prediction-Preserving Repair for Post-hoc Calibration arXiv:2609.01072v1 Announce Type: new Abstract: Post-hoc calibration corrects reported confidence, yet a multiclass calibrator can also change the associated top-1 prediction. Accuracy captures only the net effect of these changes on correctness, not how often predictions… 6 arXiv — Machine Learning research 1mo ago Bandits in Prod: Hyperparameter Optimization at Inference Time arXiv:2609.01335v1 Announce Type: new Abstract: Many production systems can assess a configuration only by using it on live requests and observing noisy feedback. Modern agentic systems are a prominent example, with inference-time choices such as model selection, retrieval… 35 arXiv — NLP / Computation & Language research 1mo ago OUTLETS: Output-Length Prediction from Speculative Decoding Backbones arXiv:2609.01068v1 Announce Type: new Abstract: The heavy-tailed distribution of output lengths in Large Language Model (LLM) serving poses major challenges for resource provisioning and cluster scheduling. Although output-length prediction can mitigate these issues, existing… 17 r/LocalLLaMA community 1mo ago Running 104GB Qwen3.8-Flash-Next on 48GB Mac at ~12 tok/s   submitted by   /u/yogthos [link]   [comments] 34 r/LocalLLaMA community 1mo ago 2/5 of my CMP 170HX have died after 2 weeks and the 3rd came with defective tensor cores. Current prices DO NOT justify the risk you are taking One GPU drops off immediately when vllm is started and the other throws CUDA errors on start   submitted by   /u/cantgetthistowork [link]   [comments] 22 r/LocalLLaMA community 1mo ago Qwen3.8-Flash-Next (104 GB MoE) on a Strix Halo + RTX 3090 Ti eGPU: 22 -> 84 tok/s, and within one HumanEval+ problem of a dual-3090 vLLM box at 0.4x the wall time Follow-up to my Qwen3.8-27B post. This time the target is Qwen3.8-Flash-Next: 512 experts per layer, 36 layers of gated DeltaNet, 12 layers of top-k sparse attention, a 26.8 GiB n-gram table and a built-in MTP draft head. unsloth UD-Q4_K_XL, 103.69 GiB. It fits in the Strix… 30 r/LocalLLaMA community 1mo ago Gave a try to Exllamav3 and it's great! Following this post I decided to try GLM 5.3 Flash on a 8x3090 setup and I can now run a Q4 with surprising speed; 700tk/s prefill & 42tk/s decoding! (lcp & vllm do not allow me to get that). Was afraid about quality but > 30m tokens with DSH and no issue (did not test vision… 36 r/LocalLLaMA community 1mo ago How I got 280 tok/s on Qwen3.8 27B on 2xr9700's and 940k tokens kv cache 2 Months ago I had made a post how I was working on my dual R9700's. It's wild to look back at where we were then and where things now stand. Since then after many users commenting and complaining about developers doing the same thing. I threw out a discord link and expected… 25 Page 6 of 10 · 500 articles ← Newer Older →
