News / #inference Tag Inference 500 articles archived under #inference · RSS Sign in to follow r/LocalLLaMA community 1mo ago Qwen 3.8 27B is faster than expected i ran this model on my two 5060 TI 16GB cards at Q4 in unsloth and LM studio ( i downloaded NVFP4 but didn't try it in vLLM ) i think it runs faster than expected it gives me 50 - 60 t/s with MTP. this is surprising because it's a dense model and Qwen 3.6 was giving me 30t/s… 34 Hugging Face Daily Papers research 1mo ago DumpsterCluster: From Dumpster Diving to Serving LLaMA-70B on $60 GPUs Abstract Retired GPUs can form low-cost clusters for LLM inference, but their economic and environmental viability depends heavily on local electricity prices and carbon intensity. Generated by thinkingmachines/Inkling-Small As AI datacenters retire functional GPUs, vast… 9 arXiv — Machine Learning research 1mo ago Forward Pass Domain Adaptation (Without Cross-Layer Backpropagation) arXiv:2608.14563v1 Announce Type: new Abstract: Forward-Pass-Only MLP training (FPO) adapts large language models without a backward pass through the model body, achieving 2.7--3.2x the throughput of standard fine-tuning at ~40% less peak training memory, while leaving… 16 arXiv — Machine Learning research 1mo ago DumpsterCluster: From Dumpster Diving to Serving LLaMA-70B on $60 GPUs arXiv:2608.14614v1 Announce Type: new Abstract: As AI datacenters retire functional GPUs, vast quantities of still capable accelerators enter secondary markets. This paper investigates whether these retired GPUs can find a productive afterlife to form a DumpsterCluster that can… 16 arXiv — Machine Learning research 1mo ago FedImp: Enhancing Federated Learning Convergence with Impurity-Based Weighting arXiv:2608.14654v1 Announce Type: new Abstract: Federated Learning (FL) is a collaborative paradigm that enables multiple devices to train a global model while preserving local data privacy. A major challenge in FL is the non-Independent and Identically Distributed (non-IID)… 30 arXiv — Machine Learning research 1mo ago Quantifying Depth Sufficiency in Residual Neural Networks: A First-Order Criterion arXiv:2608.14664v1 Announce Type: new Abstract: How can we determine whether a trained neural network is already deep enough? We study this under a fixed function-preserving residual-growth protocol specifying insertion locations, residual families, zero-output initializations,… 21 arXiv — Machine Learning research 1mo ago Probability-Preserving Transformer for the Time-Dependent Schr\"odinger Equation arXiv:2608.15112v1 Announce Type: new Abstract: Solving the time-dependent Schr\"odinger equation (TDSE) via traditional numerical methods is computationally intensive. Transformer models offer a compelling alternative, but standard implementations rely on soft constraints that… 9 arXiv — Machine Learning research 1mo ago Spectral Saliency for Machine Unlearning arXiv:2608.15548v1 Announce Type: new Abstract: Machine unlearning (MU) aims to remove the influence of specific training data while preserving model utility. As the name suggests, MU can be viewed as the inverse of learning, using gradient-based updates to reduce the influence… 25 arXiv — NLP / Computation & Language research 1mo ago Aborted but Not Forgotten: KV-Cache Retention Breaks Rollback Consistency in Language Agents arXiv:2608.15939v1 Announce Type: new Abstract: Stateful language agents assume a rejected branch can be taken back by clearing it from the application transcript. We show this breaks when the serving session retains key/value (KV) state across the logical abort: the model can… 28 llama.cpp releases dev-tools 1mo ago b10456 sycl: fix thread/block count in quantized cpy kernel launches ( #27160 ) Adjusts the thread/block count to be proportional to the size of the quant, reducing under/over subscription. Largest perf improvement is the q4_0 -> f32 path, with, on a Arc 70, throughput goes from 20.21… 10 arXiv — Machine Learning research 1mo ago CutClean: Neural Network Pruning for Privacy-Preserving Inference arXiv:2608.13773v1 Announce Type: new Abstract: Neural networks are increasingly deployed in high-stakes applications with growing privacy leakage concerns. We show that this privacy leakage can occur even in the absence of representation imbalances that lead to traditional… 29 arXiv — NLP / Computation & Language research 1mo ago QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction arXiv:2608.13966v1 Announce Type: cross Abstract: As large language model inference shifts toward lower precision, post-training quantization (PTQ) becomes increasingly brittle, making quantization-aware training (QAT) essential for preserving model quality. However, QAT… 37 arXiv — Machine Learning research 1mo ago Adjacency-Based Spectral Proxy Control of Mobile Communication Agents arXiv:2608.13616v1 Announce Type: cross Abstract: We consider a heterogeneous mobile-agent network composed of uncontrolled task agents and controllable communication agents. The objective is to reposition communication agents online as task agents move. Since throughput-based… 24 arXiv — NLP / Computation & Language research 1mo ago A Survey of Large Models in Sports arXiv:2608.14377v1 Announce Type: new Abstract: Sports have witnessed growing global enthusiasm in recent years, serving as a vital force for physical health, cultural exchange, social connection, and economic growth. The rapid advancement of large models, particularly… 24 arXiv — NLP / Computation & Language research 1mo ago Kalypso: Relational LLM Serving arXiv:2607.23815v2 Announce Type: replace-cross Abstract: Large language models are increasingly used as semantic operators for filtering, extracting, ranking, joining, and transforming unstructured data. Existing semantic query processing systems invoke request-centric LLM… 29 r/LocalLLaMA community 1mo ago Qwen 3.8 2.4T at 288k tokens/s on Nvidia GB300 NVL72 https://developer.nvidia.com/blog/serve-qwen3-8-2-4t-a95b-a-2-4t-parameter-model-with-configurable-reasoning-on-nvidia-gb300-nvl72/ 4k tokens per second per GPU of which there are 72. 350 tokens per second per user "Without additional model tuning, the model achieves a… 13 Hacker News — AI on Front Page community 1mo ago Tell HN: Cloudflare silently injects its analytics when you switch nameservers A few hours ago I switched my nameservers to Cloudflare in order to enable R2 bucket serving through my own subdomain, and I found out that it silently had injected a JS analytics snippet in my HTML-only JS-free site textlog.cc — I had to go to the Analytics dashboard, Add the… 22 r/LocalLLaMA community 1mo ago If you are at the lowest budget, which you can think of.Which hardware would you recommend to run? qwen 3.8 27b oWith like 50 tokens per second. I currently have a RTX 5070 Ti.   submitted by   /u/InternationalGap3698 [link]   [comments] 17 r/LocalLLaMA community 1mo ago Anyone managed to get Qwen 3.8 27B running smoothly on vLLM? Can't get rid of endless thinking Title pretty much says it all. I’ve deployed Qwen 3.8 27B using vLLM on an RTX 6000 Pro (tried multiple vLLM releases and launch recipes), but I can't get it into a usable state because of crazy long reasoning passes. Regardless of the thinking effort setting (xhigh, medium, or… 22 r/LocalLLaMA community 1mo ago 5090: Windows or Linux for Qwen3.8.27b I've got a dedicated AI rig sitting here with a RTX 5090 and 96GB RAM and for the past few years have been using Windows 11 and primarily LM Studio, but have also used vLLM, llama.cpp and Ollama. With Qwen3.8.27b I want to get the most out of this model. I get the feeling from… 14 r/LocalLLaMA community 2mo ago Deepseek v4 flash Q2 on a single 4090 😅 It freaking worked lol🔥 Deepseek-v4-flash-0731 @UnslothAI 's IXQ2/Q3 checkpoint on one single RTX4090 with just 64 GB of RAM at usable token rate without dspark. All kernels running on Blaze (my custom developed ML compiler + inference engine) - no llama.cpp or vllm in the… 23 r/LocalLLaMA community 2mo ago Building an open-source control plane for self-hosted vLLM, what would you want in it? Every time I self-host a model I rebuild the same stuff: start the container, set up a route, check why it died overnight, remember to shut the GPU off before it burns money. So I'm building a panel that handles it. Start/stop models, OpenAI-compatible endpoint, health checks… 25 r/LocalLLaMA community 2mo ago Try out this "high" reasoning mode for 27B (tested on VLLM) After a lot of tweaking, I have come to the conclusion that 27B lacks a reasoning mode that is between low and xhigh. The "medium" mode isn't actually medium, it erases the explicit instructions to the model. When medium is enabled, the model acts very differently - to me it… 20 r/LocalLLaMA community 2mo ago Ninfer-3090 Hi all, Of course I had to jump on Qwen3.8-27B and get it running on the RTX 3090 port of NInfer . For anyone unfamiliar with it, NInfer is a small, dedicated C++/CUDA inference runtime built around fused kernels rather than a huge general-purpose serving stack. The 3090 fork… 30 r/LocalLLaMA community 2mo ago SGLang support for Qwen3.8-27B: 200+ tok/s on 5090, 38 tok/s on DGX Spark (NVFP4 + DSpark) Hey r/LocalLLaMA 👋 This is Kai from SGLang. We just shipped day-0 support for Qwen3.8-27B. To push performance for running this model locally, we combined NVFP4 + DSpark and got: 200+ tok/s decode on a single RTX 5090 and RTX Pro 6000 38 tok/s decode on DGX Spark Here's the… 21 r/LocalLLaMA community 2mo ago NInfer day0 support for Qwen3.8 27b: ~200 tok/s generation, with tons of engine improvments Qwen3.8-27B is finally here, and NInfer already has Day-0 support! Weights: https://huggingface.co/neroued/Qwen3.8-27B-NInfer Just update to the latest source and give it a try. On a single RTX 5090, NInfer can still reach around 200 tok/s generation with speculative decoding.… 5 r/LocalLLaMA community 2mo ago Are we getting Qwen 3.8 35-A3B? So far, it seems like Qwen 3.8 might drop today as a 27B dense model. If that’s the case, no MoE offloading this time , I used to run Qwen 3.6 35B-A3B at around 70 tok/s on an RTX 3060, but offloading a dense model is a completely different story it can be 100× slower or Even… 29 Hugging Face Daily Papers research 2mo ago Knowing When to Quit: Diagnosing and Training LLMs to Abort Futile Reasoning Abstract CaRL uses reinforcement learning with refusal incentives and hindsight augmentation to reduce futile reasoning in large language models while preserving task performance. Generated by thinkingmachines/Inkling-Small Large language models generate computationally… 28 arXiv — Machine Learning research 2mo ago Structure-preserving uncertainty quantification for GENERIC dynamics arXiv:2608.12624v1 Announce Type: new Abstract: Structure-preserving machine learning embeds physical structure directly into model architectures, yet uncertainty quantification (UQ) for such hard-constrained models remains limited because standard UQ methods may violate the… 18 arXiv — NLP / Computation & Language research 2mo ago TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimes arXiv:2608.13057v1 Announce Type: cross Abstract: In expert-parallel (EP) MoE serving, every layer synchronizes at the slowest GPU. Dispatchers balance token counts (EPLB, LPLB, UltraEP) or activated-expert counts (METRO), assuming expert time is linear in one. Measurements on… 32 r/LocalLLaMA community 2mo ago Qwen 30b MoE - 30tps - 6GB vram - Done! So, I have been dreaming of getting 17 tokens per second using my RTX 3050 6GB version on a decent context window for Hermes needed above 60k. The hope is that has was a 22GB of DDR 4, hoping they can take some of those experts and give me room for context. What did I get 10 or… 33 r/LocalLLaMA community 2mo ago EXPERIMENT: Qwen3.8-2.4T-A95B running locally on an RTX 5090 + RTX 5060 Ti at ~0.80 tok/s I managed to get Qwen3.8-2.4T-A95B running locally with llama.cpp on mu PC just for fun, cause why not. I was using the Unsloth Qwen3.8-2.4T-A95B-UD-Q1_0 GGUF quantization. The full GGUF is about 397 GiB . The model uses 512 routed experts, with 10 active per token. My hardware:… 31 OpenAI official-blog 2mo ago Previewing Ultrafast mode: GPT-5.6 Sol at up to 14X the speed Preview Ultrafast, a new OpenAI API service tier that runs GPT-5.6 Sol up to 14× faster. Powered by Cerebras, it delivers up to 750 output tokens per second. 24 arXiv — NLP / Computation & Language research 2mo ago Total Recall at What Cost? Benchmarking the Serving Cost of Agentic Memory Systems arXiv:2608.11879v1 Announce Type: new Abstract: Long-running conversational agents increasingly rely on a memory system to avoid resending the whole conversation each turn, yet how much that costs to serve has received little systematic benchmarking. We compare three memory… 15 arXiv — NLP / Computation & Language research 2mo ago QV-PIC: Query-Aware Visual Position-Independent Caching for Efficient RAG Serving arXiv:2608.12121v1 Announce Type: new Abstract: Retrieval-Augmented Generation (RAG) repeatedly prefills identical text chunks across queries, incurring redundant computations. Position-Independent Caching (PIC) mitigates it by reusing precomputed Key-Value (KV) across… 27 Hugging Face Daily Papers research 2mo ago NeuPAT: Neuron-aware Plasticity Allocation Tuning for Language-Preserving MLLMs Abstract NeuPAT selectively constrains updates to language-sensitive neurons during multimodal tuning to preserve LLM language capabilities while enabling perceptual adaptation. Generated by thinkingmachines/Inkling-Small Multimodal expansion of large language models (LLMs)… 18 r/LocalLLaMA community 2mo ago Qwen 3.8 27B — MTP or DFlash? Do we.know whether the 27B model will ship with a DFlash or MTP head? It's super exciting, but since 35B-A3B is my daily driver, 27B will crawl — still excited for it though! I think 3.6 27B with MTP was about 8 tok/s for me (32GB unified memory, 780M)   submitted by  … 25 Hugging Face Daily Papers research 2mo ago SkillZip: Evaluation-Free Skill Compression for Self-Evolving Agents by Discovering Reusable Structure Abstract SkillZip compresses self-evolving agent skills by finding a minimal faithful structural explanation that shares repeated rules and procedures while preserving rare exceptions, without requiring evaluation rollouts. Generated by thinkingmachines/Inkling-Small… 26 r/LocalLLaMA community 2mo ago Meta's Muse Glimmer 30B now runs up to ~3.3x faster on Mac with mlx-dspark Been tinkering with speculative decoding on Apple Silicon for a while, and this week I got Meta's new Muse Glimmer 30B working in my project mlx-dspark . On my M4 Pro, the 8-bit model goes from 8.2 tok/s to 18-26 tok/s depending on content. Math is the best case at 3.27x, code… 12 r/LocalLLaMA community 2mo ago Benched a 124B on one DGX Spark for a week and published all of it — 38.7 tok/s on the fastest path he found, 2.4x DeepSeek V4 Flash on the same box sudoingX on X spent a week on this and posted his wrap-up. The hardware is his. He isn't affiliated with us and we didn't see any of it before he put it out — I work on Ling at inclusionAI. Where he landed on one Spark: 38.7 tok/s on the official INT4 once it's configured right,… 9 Hugging Face Daily Papers research 2mo ago UniMoMo: Expert Merging-Based MoE Acceleration for Large Recommendation Models Abstract UniMoMo compresses trained recommendation mixture-of-experts models into smaller standard MoE checkpoints via functional similarity grouping and layer-adaptive protection, preserving accuracy while accelerating inference. Generated by thinkingmachines/Inkling-Small… 27 arXiv — Machine Learning research 2mo ago Invertible Logits Transformation for Accuracy-Preserving Post-Hoc Uncertainty Calibration arXiv:2608.10372v1 Announce Type: new Abstract: Post-hoc calibration aligns a classifier's predicted confidences with its empirical accuracy without retraining. An ideal calibrator should correct nonlinear miscalibration, scale gracefully to large label spaces, and preserve the… 25 arXiv — Machine Learning research 2mo ago Benchmarking Time Series Generation Methods for Privacy-Preserving Forecasting arXiv:2608.10891v1 Announce Type: new Abstract: Time series forecasting in privacy-sensitive domains often requires training models on released data rather than original observations. Synthetic time series generation has been developed primarily for data augmentation, where… 6 arXiv — Machine Learning research 2mo ago Two-stage Odd Residual Flows for Mean-Preserving Probabilistic Time Series Forecasting arXiv:2608.11114v1 Announce Type: new Abstract: Probabilistic forecasting plays an essential role in risk-sensitive decision-making, particularly in long-horizon settings. However, existing approaches often face a fundamental trade-off between distributional flexibility and… 31 TechCrunch — AI news-outlet 2mo ago Brad Lightcap, OpenAI’s longtime COO, is leaving to ‘start something new’ One of OpenAI's longest serving executives is headed out the door, although the longtime COO told staff that he was "excited to help you all advance the mission from a different vantage point." 10 r/LocalLLaMA community 2mo ago Ling-3.0-flash quant ladder on one DGX Spark: the whole thing sits in a 32 to 40 tok/s band The interesting part of this one isn't the top number, it's how little distance there is between the top and the bottom of the ladder. Where it comes from: I work on Ling at inclusionAI, these aren't my numbers. sudoingX on X benched the full community GGUF ladder on his own DGX… 36 r/LocalLLaMA community 2mo ago Muse Glimmer 30B running locally in-browser with custom WebGPU kernels at ~25 tok/s on an M4 Max (same speed as llama.cpp)   submitted by   /u/xenovatech [link]   [comments] 23 r/MachineLearning community 2mo ago Planning/RL for a stochastic single-player merge puzzle: afterstates, previewed chance events, and long-horizon throughput [D] I am working on an AI for a small single-player merge puzzle and would appreciate pointers to related algorithms, papers, or existing implementations. It resembles 2048 in its action -> afterstate -> random event structure, but has a larger action space, stack constraints, and a… 18 Hugging Face Daily Papers research 2mo ago OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching Abstract OasisKV improves LLM inference throughput by storing full KV caches in lower memory tiers and prefetching only relevant entries into HBM using speculative-decoding lookahead predictions. Generated by thinkingmachines/Inkling-Small Large language model (LLM) inference… 6 arXiv — Machine Learning research 2mo ago ZeroLock: Concurrent Memory-Efficient LLM Training via Modular Update Decoupling arXiv:2608.07974v1 Announce Type: new Abstract: Large language model (LLM) fine-tuning at the edge adapts the model to scenario-specific data while preserving privacy. Although existing studies proposed pipeline parallelism to address the limited memory and computing resources… 10 Page 9 of 10 · 500 articles ← Newer Older →
