News / #inference Tag Inference 500 articles archived under #inference · RSS Sign in to follow r/LocalLLaMA community 19h ago Qwen 3.8 27B at 50 tok/s with 100k Context on a 16GB GPU! (beellama.cpp) I wanted to share my successful setup for running a Qwen 3.8 27B model with a massive context window on a consumer 16GB GPU (RTX 4070 Ti SUPER). The goal was to fit everything into VRAM without sacrificing quality or speed. 🧠 Key Components Model:… 38 r/LocalLLaMA community 1d ago use llms to auto annotation your dataset locally hi i make tool for this called llmog it's purpose to make llms free to - auto annotation datasets - reclassification existing yolo datasets running totally local using llama cpp or vllm or use external api you'd rather click than code. 🔗 GitHub: mohamed-em2m/llmog: framework… 36 r/LocalLLaMA community 1d ago Today I hit 181 toks/s (aggregate) on Qwen3.8-Flash-Next on 2x DGX Sparks Hey all, and hello fellow DGX Spark-ers! Today I managed some pretty crazy numbers: 181 tok/s aggregate on 2× DGX Spark on Qwen3.8-Flash-Next at 512Kcontext (2.8M kvc) I hit 181 tok/s aggregate today across a multi-agent fleet on a 2-node DGX Spark cluster. Single-stream decode… 22 NVIDIA Developer Blog official-blog 1d ago Deploy an Open Model from Checkpoint to Inference in Two Commands with NVIDIA TensorRT Model Connect Open AI models are evolving faster than ever, but bringing them into native applications can still require model-specific conversion, preprocessing,... 22 Hugging Face Daily Papers research 2d ago Thinking on Shots: Consistent Multi-Shot Video Editing with Agentic Reasoning Abstract An agentic framework combining LLMs and VLMs enables consistent, multi-instruction editing of long multi-shot videos while preserving spatiotemporal structure. Generated by thinkingmachines/Inkling-Small While generative AI has significantly advanced video editing,… 16 arXiv — Machine Learning research 2d ago Predicting Quantifiability from Primary Screens to Prioritize Dose-Response Profiling arXiv:2608.26538v1 Announce Type: new Abstract: High-throughput drug screening relies on low-cost primary assays to prioritize compounds for more expensive dose-response profiling, where potency is ultimately quantified. Current screening strategies largely focus on identifying… 20 arXiv — NLP / Computation & Language research 2d ago Knowledge-Verified Emergent Deception in LLM Agents Under Conflicting Incentives arXiv:2608.26372v1 Announce Type: new Abstract: Large language models are increasingly deployed as autonomous agents serving users on behalf of companies, placing them in settings where user and deployer interests can conflict. When an agent knows that a user is owed something… 23 arXiv — NLP / Computation & Language research 2d ago Preserving General Capabilities during Domain Specialization with Uncertainty-Calibrated MOPD arXiv:2608.26735v1 Announce Type: new Abstract: Specializing large language models to vertical domains improves domain-specific behavior but often degrades general capabilities such as reasoning, coding, instruction following, and creative writing. We study this domain--general… 29 r/LocalLLaMA community 2d ago Ninfer and a 5090 with 3.8 27B is making me cry tears of joy it's so good. Built the latest and I'm getting as much as 220 tokens per second and averaging in the 170s, I can't get over it. If anyone on here is on that project, fuckkkin' chapeau man, really incredible job. I can't believe I was able to like double or more my throughput from llama.cpp… 34 r/LocalLLaMA community 2d ago GLM-5.3-Flash @ DGX Station GB300: ~206 tok/s (single stream), 1M context Hey all! I'm finally doing some cool stuff with my "thinking heater" (h/t u/-TV-Stand- ). I'm still experimenting with GLM-5.2 (in anticipation of 5.3 coming tomorrow, I hope!) and things are very cool so far. With the release of GLM-5.3-flash, I decided to play with it on the… 25 arXiv — Machine Learning research 3d ago ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Acceleration arXiv:2608.24938v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) models scale capacity for strong quality while keeping per-token compute bounded through sparse expert activation. Yet low-latency MoE serving is increasingly challenging, because it spans two inference… 9 arXiv — Machine Learning research 3d ago NVExplain: Explaining Time Series Forecasting with Latent Trajectory Analysis and Structure-Preserving Surrogates arXiv:2608.25080v1 Announce Type: new Abstract: Time series forecasting models are widely used in high-stakes settings, yet their predictions remain difficult to interpret because existing post-hoc methods often ignore temporal dependence and fail to provide horizon-specific… 20 arXiv — Machine Learning research 3d ago Transforms for LLM Quantization: The Great Inversion and Format Co-Design arXiv:2608.25188v1 Announce Type: new Abstract: Most competitive 4-bit LLM research pipelines now open the same way: apply a linear, function-preserving transform (rotation, scaling, permutation, non-orthogonal affine) so the outlier mass sits more favorably against the group… 12 arXiv — Machine Learning research 3d ago Cooperative Multi-Agent Reinforcement Learning for Adaptive Aggregation in Semi-Supervised Federated Learning with non-IID Data arXiv:2608.25794v1 Announce Type: new Abstract: Federated Learning (FL) enables distributed training of machine learning models while preserving data privacy. However, FL struggles with heterogeneous, non-IID client data distributions, resulting in sub-optimal and biased global… 19 arXiv — Machine Learning research 3d ago When Pruning Meets Interpretability: Preserving Sparse Autoencoder Robustness in LLMs arXiv:2608.25941v1 Announce Type: new Abstract: Sparse autoencoders (SAEs) are widely used to interpret the internal representations of large language models (LLMs), yet their reliability under post-hoc model compression remains poorly understood. We present a systematic study… 17 arXiv — NLP / Computation & Language research 3d ago Less can be More: Relieving RAG Bottlenecks via Evidence Frontloading and Pressure-Adaptive Budgeting arXiv:2608.25115v1 Announce Type: new Abstract: Existing methods for improving Retrieval-Augmented Generation (RAG) efficiency mainly optimize downstream LLM generation, such as context compression or serving optimization. However, RAG is an end-to-end system, and its bottleneck… 29 arXiv — NLP / Computation & Language research 3d ago TOPAS: Workflow-Aware Prefix-State Scheduling for Multi-Agent LLM Serving arXiv:2608.25523v1 Announce Type: new Abstract: Prefix caching introduces a fundamental tradeoff in multi-agent large language model (LLM) serving: retaining a long system-prompt key-value (KV) cache for an agent accelerates future calls, yet it reduces the GPU memory available… 5 r/LocalLLaMA community 3d ago Qwen3.8 27B C8 at 972 TG / 5,680 PP on 4x MI100 rig ($6.5k) using my new INT8 vLLM fork Yet another vLLM fork thread here, but this time its for older INT8-centric hardware. This is a complete INT8 serving stack for Qwen3.8 27B based on vLLM, AITER, and a 27B GPTQ INT8 quant w/ DFlash2 . Its not just another vibed autoresearch loop. No, vLLM ships with very little… 9 r/LocalLLaMA community 3d ago Lemonade end-of-summer project update, now serving 15 engines! Hi everyone, it's been a while since I posted so here's an update on what the Lemonade community has been up to this summer. Our overall mission is to enable local AI builders with everything they need to make great apps and agents, while keeping the stack turnkey, portable, and… 23 r/LocalLLaMA community 4d ago Open Source Kernel in Qwen3.6-35B-A3B for AMD MI350X: 78,498 output tok/s on 8 GPUs So here's the thing, almost everyone use NVIDIA to run their LLMs, we also do the same, a lot of people we've met use like RTX PRO 6000 or even H100, B300 It seems like everyone eyes is looking at NVIDIA. However we do the math that the raw power alone on AMD GPU MI350X is… 9 arXiv — Machine Learning research 4d ago Calibration-Preserving Pruning: Compression as a Reliability Contract arXiv:2608.23744v1 Announce Type: new Abstract: Split conformal prediction, not the pruning rule, supplies finite-sample marginal coverage once a pruned model is fixed independently of the conformal calibration split. We study the separate efficiency problem: can pruning… 31 arXiv — Machine Learning research 4d ago PinSieve: Production Selective VLM Serving and a Governed Memory Flywheel for Enterprise Content-Quality Triage arXiv:2608.24040v1 Announce Type: new Abstract: Enterprise AI agents in production often need to be bounded, stateful, observable, and governable rather than fully autonomous. We present PinSieve, a production case study in a large-scale content-quality pipeline. Its deployed… 21 arXiv — Machine Learning research 4d ago Conditional GraphGANFed: Optimizing Graph-Structured Molecule Generation in Federated Generative Adversarial Networks arXiv:2608.24610v1 Announce Type: new Abstract: Generative adversarial networks (GANs) have garnered considerable attention in molecular discovery for their ability to generate novel and high-quality molecules. To efficiently train a GAN model while preserving data privacy,… 18 TechCrunch — AI news-outlet 4d ago OpenAI’s Jalapeño chip is built for fast inference at scale, benchmarks show Tested on SemiAnalysis’ InferenceX benchmark, Jalapeño registered both more tokens per user and more throughput per kilowatt than the currently available state-of-the art. 6 Hugging Face Daily Papers research 5d ago Beyond the Stability-Exploration Dilemma: Environmental Regularization for LLM Policy Optimization Abstract ERPO replaces action-side policy regularization with input-side query distribution control to stabilize reinforcement learning for language models while preserving response exploration. Generated by thinkingmachines/Inkling-Small Policy optimization (PO) for Large… 4 OpenAI official-blog 5d ago Jalapeño’s first results show industry-leading speed and efficiency in AI inference Jalapeño is a custom inference chip from OpenAI that delivers faster, more power-efficient AI inference, with higher throughput and lower latency for modern models. 34 Hugging Face Daily Papers research 5d ago Block3D: Efficient Text-to-3D Generation via Block-Wise Diffusion Abstract Block3D accelerates text-to-3D generation by using block-wise diffusion with confidence-guided correction to reduce inference time while preserving geometric fidelity. Generated by thinkingmachines/Inkling-Small While text-to-3D generation has advanced rapidly,… 13 arXiv — NLP / Computation & Language research 5d ago Can LLMs Truly Forget? Revealing Unlearning Gaps Through Adversarial Evaluation arXiv:2608.21606v1 Announce Type: new Abstract: Machine unlearning aims to remove the influence of targeted training data from a model while preserving its remaining capabilities, but evaluating whether such information has truly become inaccessible remains challenging. Existing… 19 Hugging Face Daily Papers research 5d ago TileMix: Tile-Centric Mixed-Precision Attention for LLM Inference Acceleration Abstract TileMix routes attention score tiles to mixed FP16 or INT8 precision within fused dense attention, recovering long-context accuracy while improving prefill throughput without retraining. Generated by thinkingmachines/Inkling-Small Long-context prefill in large language… 5 r/LocalLLaMA community 5d ago [2608.16157] FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution Source of Claims: https://x.com/Andy_ShuoYang/status/2090856976880472439 Your gaming PC can now serve frontier models at interactive speed using official checkpoints without extreme quantization! Qwen3.6 35B → 8GB RTX 4060 laptop @ 39 tok/s DeepSeek-V4-Flash 284B → RTX 5090… 22 r/LocalLLaMA community 6d ago Qwen 3.8 27B Aider score I ran the Aider benchmark on Qwen 3.8 27B FP8 with FP8 KV cache 256K context vLLM. The score: 72.9 This matches Gemini 2.5 Pro from 2025-04-12 which also scored 72.9. Beats Claude Opus 4 from 2025-05-25 which scored 72.0. DeepSeek R1 2025-06-06 scored 71.4. It may just be a… 24 arXiv — Machine Learning research 6d ago FlatLand: Personalized Graph Federated Learning via Tailored Lorentz Space arXiv:2608.21096v1 Announce Type: new Abstract: Federated learning enables privacy-preserving collaborative training, but highly heterogeneous client data remain challenging, especially in graph federated learning where clients possess structurally diverse graphs. Existing… 11 arXiv — Machine Learning research 6d ago Amplifying the imaging power of digital sky surveys with space telescopes data and generative AI arXiv:2608.20666v1 Announce Type: cross Abstract: While Digital sky surveys provide excellent throughput of image data and can cover a large footprint, their imaging power is normally inferior to that of space-based telescopes. Space-based telescopes, on the other hand, provide… 31 arXiv — NLP / Computation & Language research 6d ago SAC-Copula: Quality-Preserving Watermarking for Diffusion Language Models via Smooth Correlated Gumbel Fields arXiv:2608.20839v1 Announce Type: new Abstract: Watermarking diffusion language models (DLMs) requires mechanisms compatible with iterative parallel unmasking rather than autoregressive decoding. Existing sampling-based watermarking methods typically inject position-wise i.i.d.… 14 arXiv — NLP / Computation & Language research 6d ago Quantization-Aware Healing: A Practical Recipe for Recovering Compressed, 4-Bit LLMs arXiv:2608.20953v1 Announce Type: new Abstract: Serving large language models cheaply increasingly means shipping models that are both structurally compressed to a fraction of their parameters and quantized to 4 bits. Together these steps degrade reasoning, mathematics, coding,… 5 arXiv — NLP / Computation & Language research 6d ago Knowledge-Graph-Gated Defactualization for Style-Controllable and Fact-Preserving Generation in Agentic Conversational AI arXiv:2608.20393v1 Announce Type: new Abstract: Agentic large language models (LLMs) deployed in fact-sensitive applications such as customer support must simultaneously preserve factual correctness and generate responses in a controllable stylistic register. Activation steering… 7 arXiv — NLP / Computation & Language research 6d ago Target-Aware Calibration Data Selection for Preserving Uncertainty in Quantized Language Models arXiv:2608.21019v1 Announce Type: new Abstract: Quantization is widely used to deploy large language models, but its effect on uncertainty behavior, such as confidence, margins, and abstention, is rarely treated as a primary objective. We frame calibration-data selection for… 29 Hugging Face Daily Papers research 6d ago CLEAR: Continuous Latent Adapter Routing for Utility-Preserving LLM Safety Alignment Abstract CLEAR uses a hidden-state gate to continuously modulate a safety low-rank adapter, improving LLM safety while preserving utility on benign inputs. Generated by thinkingmachines/Inkling-Small Improving the safety of large language models (LLMs) often comes at the expense… 37 r/LocalLLaMA community 6d ago Ling Tiny, King of Speed Ling Tiny has now replaced Gemma4-12B in my rig as an auxiliary model doing hindsight operations. This is on a 4060Ti, which is a reasonable GPU available out there, and the speed is phenomenal. Don’t enable MTP, set up the vLLM fork for BailingMoE3. Hope this is useful to… 27 r/LocalLLaMA community 7d ago I hosted Kimi K3 (2.8T parameters) using 8 B300s. 92 tok/s, $190 per million tokens What I ran: 8x B300 on Modal, $56.79 per hour, vLLM, tensor parallel 8, native MXFP4 Cold boot ~27 min (1.56 TB load, JIT, 51 CUDA graph captures) TTFT 0.92 to 1.02 s, decode 92 tok/s steady, 83 tok/s average over 4 prompts $190 per million output tokens. One clean run is about… 10 r/LocalLLaMA community 7d ago i finally switched from windows to linux and got a 30-50% boost in speed. This is amazing. All I did was switch from llamacpp on windows to vllm on linux.   submitted by   /u/unraveleverything [link]   [comments] 29 r/LocalLLaMA community 7d ago Single RTX 5090: Qwen3.8-27B NVFP4 at a real 262K context in vLLM — 77 tok/s short-context, 64.7 tok/s at 128K This is the Qwen3.8-27B setup I actually use every day on one RTX 5090. I wanted to write it down with enough detail that another 5090 owner can reproduce it instead of guessing which memory knobs I used. The short version: the full 262,144-token window fits together with… 25 r/LocalLLaMA community 7d ago I forked Ninfer 3090 and converted it to run on the CMP170HX - doubled my Qwen3.6-35B from llama.cpp Good afternoon, everyone! I wanted to show the work I've been doing around porting Ninfer over to the CMP170HX ( Github ) So, first, I do want to call out the amazing work that Neroued , Sergiuszm and specifically Don-Chad have all done, to not take away from vLLM, Llama.cpp and… 37 r/LocalLLaMA community 8d ago Freetokens project is impressive A new project was released yesterday and I have the opportunity to test it today. Papper: https://arxiv.org/abs/2608.16157 Github: https://github.com/FlashML-org/FreeToken My initial tests with the following setup: RTX 5080 (16 GB) DDR6 64GB AMD Ryzen 9 9950X3D I got 100tok/s on… 14 r/MachineLearning community 8d ago I developed my own quantized LLM from scratch, trained on 30B tokens, deploys in 60 MB [R] I trained a 250M parameter model from scratch on 30B tokens of fineweb. It’s quantized to under 2 bits so the whole deployment is 60 MB and it needs about 80 MB of RAM to run. Runs around 400 tok/s on a normal laptop CPU, no GPU needed. How the long context works: the most… 10 r/LocalLLaMA community 8d ago I feel like I finally graduated. I finally made the move from LM Studio to vLLM thanks to this post https://www.reddit.com/r/LocalLLaMA/s/NmS9CgHvqz . I may not know what it all means yet but I’m going to start diving into the docs to learn as much as I can. I’m running an endpoint on each of my 3090s one for… 4 r/LocalLLaMA community 8d ago Ornith-1.5-35B-A3B-NInfer - 250 tok/s, 5-8k prefill, 5090 I tried this model yesterday, and it felt to me like the best one I've tried for a local model for interactive use; the responses and reasoning are very fast, and it actually performs agentic tasks well. The speed is phenomenal. I am running this on Ninfer for Windows -… 37 r/LocalLLaMA community 9d ago Qwen3.8-27B at 262K context on a Strix Halo + RTX 3090 Ti: 9.5 -> 153 tok/s, and it beats a dual-3090 vLLM box on HumanEval Spent a while treating layer placement, KV format and llama.cpp itself as experimental variables. 159 logged experiments. Numbers first, caveats after. Hardware: AMD Ryzen AI MAX+ 395 (Strix Halo, 128 GB unified) + RTX 3090 Ti on an eGPU link. One llama.cpp process, AMD on… 37 Hugging Face Daily Papers research 9d ago FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving Abstract FlashPrefill V2 improves long-context serving via mean-corrected sparse attention, optimized GPU operators, and framework integration, achieving large speedups over dense baselines. Generated by thinkingmachines/Inkling-Small Long-context modeling is a pivotal… 9 arXiv — Machine Learning research 9d ago FleetSieve: Decision-Critical Profiling for SLO-Aware LLM Fleet Configuration arXiv:2608.19659v1 Announce Type: new Abstract: Choosing tensor-parallel (TP) degrees and replica counts for an LLM serving fleet is difficult because performance is not monotonic in TP and the feasible choice can change with load. Exhaustive profiling resolves this uncertainty,… 38 Page 1 of 10 · 500 articles Older →
