News / #inference Tag Inference 500 articles archived under #inference · RSS Sign in to follow arXiv — NLP / Computation & Language research 6d ago Same Text, Different Prediction: Serving-Context Nondeterminism in Text Classifiers arXiv:2610.09111v1 Announce Type: new Abstract: Deterministic inference is essential for reliable and trustworthy machine learning. Prior studies of text generation have shown that changing factors such as batch size, batch composition, hardware, or inference engine can alter… 22 arXiv — NLP / Computation & Language research 6d ago Judging in Latent Space: Efficient Generative Reward Modeling via Semantics-Preserving Compression arXiv:2610.09788v1 Announce Type: new Abstract: Reward modeling often requires jointly representing and reasoning over multiple evaluation criteria, yet verbalizing this process token by token can incur substantial inference cost. Recent work on latent reasoning suggests that… 24 r/LocalLLaMA community 6d ago 2x CMP 170HX 64GB: GLM-5.3-Flash at 384K context / ~90 tok/s (EXL3, HBM-first setup) + Qwen3.8 comparison I've been tinkering with GLM-5.3-Flash on two 64GB CMP 170HX cards for a while, and the setup is finally stable enough that I figured I'd share it. I also compared it against the Qwen3.8-Flash-Next setup I've been using on the same machine: AWQ INT4 + FP8 PLE on vLLM. Besides… 18 NVIDIA Developer Blog official-blog 7d ago Faster Scientific Image Analysis with NVIDIA cuPhoton Observatories and telescopes, lasers and X-ray light sources, and other high-throughput instruments generate image data faster than CPU-bound pipelines can... 18 r/LocalLLaMA community 7d ago I re-trained the DFlash 2 drafter for Ternary Bonsai 2 27B: 2.2x on an L4 (3.2x on code edits with ngram lookup), 1.5x on a Mac, 1.2x in Chrome PrismML's Ternary Bonsai 2 27B fits a 24 GB card or Mac, but decodes at ~30 tok/s on an L4 and ~21 on an M4 Pro. z-lab's DFlash 2 drafter was trained on bf16 Qwen3.8-27B, so it guesses worse on the ternary model. I fine-tuned it on 1.5M tokens of Bonsai 2's own greedy output.… 22 arXiv — Machine Learning research 7d ago Fed-BRDECS: Privacy-Preserving and Heterogeneity-Aware Federated Deep Embedded Clustering arXiv:2610.07399v1 Announce Type: new Abstract: Federated deep clustering seeks to learn clustering-friendly representations from decentralized unlabeled data while preserving client privacy. However, Deep Embedded Clustering (DEC)-style objectives depend on global… 21 arXiv — Machine Learning research 7d ago Preserving Unstable Modes Through Inverse Dynamics in JEPA World Models arXiv:2610.07540v1 Announce Type: new Abstract: Robotic systems often exhibit unstable modes, along which small perturbations and disturbances can cause unbounded growth unless corrected through feedback. Controlling such systems from high-dimensional visual observations… 29 arXiv — NLP / Computation & Language research 7d ago APEX: Speculate smarter, not deeper arXiv:2610.07780v1 Announce Type: new Abstract: Speculative decoding reduces large language model inference latency by drafting multiple tokens before target-model verification, but its effectiveness depends on both the proposal mechanism and draft depth. Fixed configurations… 36 arXiv — NLP / Computation & Language research 7d ago Lineage-Aware Memory Governance: A Derivation-Gated Framework for Privacy-Preserving Column-Level Access Control in Enterprise AI Agents arXiv:2610.07258v1 Announce Type: cross Abstract: Enterprise AI agents that share a memory store face two unaddressed risks: sensitive data can leak through legitimately computed results the requester could not derive, and departments can silently compute a same-named key… 26 r/LocalLLaMA community 7d ago Qwen3.8-27B with just 12GB VRAM + 8GB RAM - ~18 tok/sec I got Qwen3.8-27B running at ~18 tok/sec decode & ~500-600 tok/sec prefill (at 64k context) on just 12GB VRAM + 8GB RAM, using ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S quant (~11GB). This recipe also works with 0bserverx’ Qwen3.8-27B-Heretic-GSQ-RCO IQ3_S quant, achieving similar… 29 The Information — AI news-outlet 7d ago Pinterest Names New CFO Pinterest named Amazon executive James Dibbo as chief financial officer. It’s the fourth C-suite shakeup for the social media firm this year. Dibbo, who will start at Pinterest on October 26, previously led finance at Amazon’s media and retail divisions, most recently serving as… 38 r/LocalLLaMA community 8d ago GPT-6.1 Sol looped "leak" hints at nested models serving architecture Hello llamas. I am posting this because I believe that, despite it being closed source models, the discussion will bring value to the local AI community. As many of you probably heard, GPT-6 Astra is speculated to be a looped transformer architecture that outputs a token after… 31 r/LocalLLaMA community 9d ago Qwen3.8-Flash-Next (125B) on a single Strix Halo mini PC: 44-59 tok/s with speculative decoding, ~1,400 tok/s prefill, engine is open Hey all. We've spent the last weeks getting Qwen3.8-Flash-Next (125B MoE, 6B active) to run properly on one AMD Strix Halo box (Ryzen AI Max+ 395, 128 GB). Tonight we're releasing both the 95 GB EXL3 weights and a new version of Kyojin, our inference engine (built on ExLlamaV3,… 9 r/LocalLLaMA community 9d ago M5 Ultra 256 running GLM 5.3 Flash 68.8 tok/s Running oQ4e+MTP on oMLX 0.7.0, with still more to optimize. Prefill is 1,878 toks. I saw some other benchmarks below what id expect so i figured I would share.   submitted by   /u/cryotic [link]   [comments] 37 r/LocalLLaMA community 9d ago Ok how to actually learn vLLM ? Said in title, I find the ecosystem difficult to understand, and RTFMing doesn't help me as it's never clear what is the server vs their client library ? I'm using it for voxtral 3B on one GPU, but it's because I can run that with no quantization, I'm lost on learning to run… 18 r/LocalLLaMA community 9d ago Switched my local agent from Qwen3.8 27B to Ornith 1.5 35B-A3B on two 5070 Tis: about 180 tok/s vs 60, same scores on my tests My setup is two RTX 5070 Ti 16GB cards (the second one is on an OCuLink dock) with 64GB of RAM, Ollama on Windows, and the agent runs on pi in WSL. Until last night the daily model was Qwen3.8 27B UD-Q4_K_XL at 128K with MTP, which does about 55 to 70 tok/s across both cards. I… 24 arXiv — Machine Learning research 9d ago Rank-Aware Speculative Sampling for Diffusion Draft Trees arXiv:2610.02251v1 Announce Type: new Abstract: Speculative sampling accelerates diffusion generation by verifying inexpensive draft states in parallel while preserving the target law. Recent tree-based methods allocate the parallel compute budget more effectively than… 7 arXiv — Machine Learning research 9d ago BaCP: Backbone Contrastive Pruning for Preserving Representations in Extremely Sparse Neural Networks arXiv:2610.02524v1 Announce Type: new Abstract: Unstructured pruning at extreme sparsity often suffers from representational collapse, causing sharp drops in accuracy. To address this, we study Backbone Contrastive Pruning (BaCP), which regularizes the sparse network's embedding… 28 arXiv — Machine Learning research 9d ago What Is Lost in Post-Training? Default Collapse and the Loss of In-Context Steerability Across Diverse Perspectives arXiv:2610.02614v1 Announce Type: new Abstract: AI models serving a heterogeneous population must act on the principles appropriate to each user and context. While post-training has been shown to narrow the views large language models express, prior work has focused on default… 5 arXiv — Machine Learning research 9d ago Prospective Hindsight: Self-Calibrating Reinforcement Learning via Prediction-Reality Gaps arXiv:2610.02740v1 Announce Type: new Abstract: Reinforcement learning for long-horizon agents relies on purely retrospective training signals: credit is assigned only after observing environmental consequences, leaving the agent's belief at action time invisible to the… 24 arXiv — Machine Learning research 9d ago A Controlled Audit of Personal AI Memory for Rating Prediction arXiv:2610.02764v1 Announce Type: new Abstract: In structured rating prediction, does a personal AI use historical item-rating associations, or mainly the user's rating tendencies? We audit this distinction by permuting historical ratings within each user while preserving the… 8 arXiv — Machine Learning research 9d ago Exact Memory-Time Optimization for Prefix-Cached Language Model Serving arXiv:2610.02766v1 Announce Type: new Abstract: Retaining language-model prefix states trades recomputation against storage time. Optimizing each cached block independently can overcount savings: a resident block is usable only when the required preceding prefix is also… 31 arXiv — NLP / Computation & Language research 9d ago Learning When to Commit from Partial Speech for End-to-End Simultaneous Speech Translation arXiv:2610.02612v1 Announce Type: new Abstract: Simultaneous speech translation must emit useful target text before the source is complete while preserving every committed token. We adapt a full-utterance speech language model using prefix supervision derived from its own… 15 arXiv — NLP / Computation & Language research 9d ago AptMQL-Bench: From Text-to-SQL to Text-to-MQL via Access-Pattern Schema Design and Data-Preserving Migration arXiv:2610.02770v1 Announce Type: new Abstract: Document databases such as MongoDB are core infrastructure for modern applications, and natural-language interfaces to them---text-to-MQL---would let non-experts query complex, semi-structured data without mastering the query… 16 r/LocalLLaMA community 10d ago Fully local little parkour sim I vibed this up this weekend, fully local, with GLM 5.3 Flash running on 2x DGX Sparks. vllm TP2 recipe: https://github.com/tonyd2wild/GLM-5.3-Flash-NVFP4-DFlash2-2x-DGX-Spark Prefill: ~1500t/s Decode: ~40t/s @ 100k Using Claude Code as the scaffold with 260k context size. I'm… 31 r/LocalLLaMA community 10d ago poorman inference engine for 16GB GPU and 35B moe Qwen 3.6for coding I forked llama.cpp's server into AgrillaMoE, a dedicated build for Qwen3.6-35B-A3B (~A4B) with Unsloth quants. On a (vant.ai) rented V100 16GB with the 2-bit UD-Q2_K_XL quant it generates at ~57-60 tok/s while running the full MoE-expansion profile — and it speaks both the… 7 r/LocalLLaMA community 10d ago [Discussion] A 5KB pure x86-64 assembly engine for Gemma-2B (FP16, 4.6 tok/s on CPU) Hi everyone, Sharing a personal project exploring the minimal bare-metal footprint required to run an autoregressive LLM. Instead of relying on large runtimes or compiler abstractions, I wrote an inference engine for Gemma-2B entirely in flat x86-64 assembly (FASM): - **Binary… 37 r/LocalLLaMA community 11d ago I built Ninfer 4080 for 16GB class GPUs Hi everyone, TL/DR I created NInfer 4080 to run ISTA-DASLab-Qwen-3.8-27B-GSQ at 100k context on an RTX 4080 16GB GPU using way more of the hardware capabilities ( max overall: 2720 tok/s prefill, 262 tok/s generation ) and sharing it with the community now so others can also… 6 r/LocalLLaMA community 11d ago The Rise of Overfit Inference Engines There seems to be a whole category of extremely narrow inference runtimes appearing: Strata, ninfer, DwarfStar, Splash, llamAmpere, gufo, etc. They deliberately give up the thing llama.cpp/vLLM are great at - generality - and optimize around a small number of models and… 34 r/LocalLLaMA community 11d ago Two ~300B MoE models, each on ONE 128 GB mini PC (AMD Strix Halo): GLM-5.3-Flash at ~580 tok/s prefill, MiMo-V2.6-Flash up to 44 tok/s decode. EXL3 weights + open ROCm engine We built an engine, Kyojin, on top of ExLlamaV3 for Strix Halo (gfx1151, ROCm), and packed two 300B-class MoE models so each fits one 128 GB machine. First release, all measured on Ryzen AI Max+ 395. Model GLM-5.3-Flash MiMo-V2.6-Flash-MOPD Size 99.7 GB 105 GB Prefill 580 tok/s… 19 r/LocalLLaMA community 11d ago Qwen3.8-Flash-Next 177B running at 11–15 tok/s on a single RTX 5070 12GB + 32GB RAM DDR4 Benchmarking an LLM here with a NVIDIA RTX 5070 12 GB VRAM here I had been working on a llama.cpp based expert streaming setup for Qwen3.8-Flash-Next 177B (UD-IQ3_XXS) on Windows. Benchmark is about 11.5 tok/s, up from roughly 7 tok/s on the inherited setup. In normal… 35 r/LocalLLaMA community 11d ago gufo-Qwen3.6-35B-A3B-Q6dense - 3095tok/s prefill; 190 tok/s decode on Strix Halo   submitted by   /u/nubela [link]   [comments] 6 arXiv — Machine Learning research 11d ago Denoising Surface: Modeling and Predicting Inference Cost for Diffusion LLM Serving arXiv:2610.00499v1 Announce Type: new Abstract: As diffusion large language models (dLLMs) become more capable, they are moving from research settings to real-world \textit{serving}, where request management (such as scheduling and resource allocation) relies on accurate… 23 arXiv — Machine Learning research 11d ago Geometry-Dependent Bounds for Online Non-Monotone DR-Submodular Maximization arXiv:2610.00545v1 Announce Type: new Abstract: We study adversarial online maximization of nonnegative, non-monotone DR-submodular functions over compact convex down-closed sets. A learner commits each action before observing its objective and competes with the best fixed… 22 r/LocalLLaMA community 12d ago I spent 3 weeks testing local Qwen3.8 on the new low-latency SGLang/vLLM recipes: DFlash2 2.8x. Builds: RadixArk + Inferact 27B NVFP4, 27B BF16, orcarouter 27B Uncensored, Flash-Next NVFP4 Hey guys, Last time I tested Qwen3.8-Flash-Next on its own. This time I put three Qwen3.8 checkpoints through the same 10 tests on the same RTX PRO 6000: RadixArk/Qwen3.8-27B-NVFP4 (dense) orcarouter/Qwen3.8-27B-Uncensored-NVFP4 (dense, uncensored fine-tune)… 20 r/LocalLLaMA community 12d ago Strata on a power limited 5090 and 96GB of DDR5-6400 is cranking out 150-200 tok/s decode and 5-6k prefill! Qwen3.8-Flash-Next at IQ3_S, CTX at 128k tokens (8-bit).   submitted by   /u/z0_o6 [link]   [comments] 10 r/LocalLLaMA community 12d ago Two 96 GB Ascend cards crun Qwen3.8-flash-next hardware notes, vLLM work, benchmarks, and what is next I have been building a somewhat unusual local inference machine around two Huawei Atlas 300I Duo cards. They are relatively inexpensive, passive, dual-accelerator PCIe cards with 96 GB of device memory apiece. They are also absolutely not drop-in CUDA replacements. When I first… 36 arXiv — Machine Learning research 12d ago Continual Learning of Dynamical Systems in Recurrent Neural Networks through Recyclable Unit Gating arXiv:2609.38356v1 Announce Type: new Abstract: Dynamical Systems Reconstruction (DSR) aims to infer models from observed time series that reproduce a system's qualitative long-term behavior. Continual DSR (cDSR) requires learning new systems while preserving previously learned… 18 arXiv — Machine Learning research 12d ago Generalized Residual Closure: General Learning Dynamics for Stability-Plasticity Compatibility arXiv:2609.38911v1 Announce Type: new Abstract: Learning must acquire new capabilities while preserving both prior responsibilities and the capacity to learn again. We introduce Generalized Residual Closure (GRC), a framework for learning as recursive closure of future-relevant… 30 r/LocalLLaMA community 12d ago Update on the “Monstrosity”. 6 BC-250 board cluster This is 6 bc-250 ex mining boards with 5 in the asrock 4u12g case they came in. After a lot of testing my current preferred setup is 4 boards running Qwen Next Flash IQ2_XS at 100k context with around 28 tok/s for short generation and 24 tok/s at 50k with around 115 ppt. The… 15 r/LocalLLaMA community 13d ago Gufo performance .... 70tps Qwen 3.8 27b but you need to read the fine print. So everyone has been yelling about how I should be using Gufo instead of halogen because it's open source and it's "just as good or better". Checking in on their GitHub (GitHub.com/gufo-org/gufo) got me immediately .. "Qwen 27B Q4: 70.56 tok/s single user, 123 tok/s with 8… 19 r/LocalLLaMA community 13d ago Running 95.5 GiB Qwen3.8-Flash-Next at 41–52 tok/s on a 64GB Mac (1.76x faster than llama.cpp): Slipstream release, 130k context scaling, + Swift variant I've been working on getting the 95.5 GiB Qwen3.8-Flash-Next model to run fast on a single 64GB Mac. In my earlier posts, I shared a custom expert-streaming fork of llama.cpp . It worked, but decode capped out around ~23–27 tok/s and slowed down as context grew. Today I'm… 5 NVIDIA Developer Blog official-blog 13d ago Build Local AI Apps with C++ and NVIDIA TensorRT RTX Samples Adding AI models to local applications requires a portable model format, a reliable runtime, and acceleration that works across target systems. Do Inference Now... 22 r/LocalLLaMA community 13d ago $5400 eBay 8x V100 server cranks on flash-next Had opus 5.5 do the bring up.. super solid results. Using only 4 GPU results in KV of like 120k with image on. 27B TP=4 results in >200 tok/s with dflash, prefill around 2.5-3.5k. Both these are running the nvfp4 Nvidia checkpoints. Found a magic repo that unpacks into fp16 on… 22 r/LocalLLaMA community 13d ago Dual GPUs vLLM resources for 50 + 40 nvidia GPUs i wanted to see how far i can push my PC and on the way i discoverd the resources are limited so i made my own and i hope it helps . https://github.com/ExTV/dual-gpus-vllm   submitted by   /u/Fz1zz [link]   [comments] 30 arXiv — NLP / Computation & Language research 13d ago Personalized State-Transition-Aware Memory for Clinical Agents arXiv:2609.38490v1 Announce Type: new Abstract: Large language model (LLM) agents that reason over clinical records must track changes in a patient's state while preserving the history needed to understand them. Simply accumulating memories leaves it unclear which information… 11 arXiv — NLP / Computation & Language research 13d ago Strong Multilingual Privacy Tagging at Encoder Speed arXiv:2609.38630v1 Announce Type: new Abstract: Privacy redaction must remove personal information while preserving relationships expressed in text. We develop a multilingual named-entity tagger with fine-grained distinctions supporting varied redaction policies and methods for… 7 arXiv — NLP / Computation & Language research 13d ago RAIM: Robust Aggregation of Inexpensive Models for Hallucination Detection arXiv:2609.39229v1 Announce Type: new Abstract: Automatic evaluation of faithfulness increasingly relies on a large language model acting as a judge, yet the most reliable judges are proprietary frontier models, costly and ill-suited to high-throughput monitoring. We investigate… 6 r/LocalLLaMA community 13d ago tp=6 can work on vLLM, with padding vLLM's tensor parallel requires that several of the model architecture numbers be evenly divisible by the number of GPUs selected for tensor parallel. This usually means that you can only use a number of GPUs that is a power of two (2, 4, 8, etc.). I have six GPUs, and I want to… 8 Google DeepMind official-blog 14d ago Introducing SynthID Bio Proof of concept for watermarking AI-generated proteins while preserving biological function. 19 Page 2 of 10 · 500 articles ← Newer Older →
