News / #inference Tag Inference 500 articles archived under #inference · RSS Sign in to follow arXiv — Machine Learning research 2mo ago Efficient Test-Time Scaling for LLM-based Time Series Forecasting arXiv:2608.08675v1 Announce Type: new Abstract: Long-term time series forecasting benefits from preserving global structure such as trends and seasonality. Recent LLM-based forecasters often improve accuracy through test-time scaling (e.g., iterative refinement), but these… 27 arXiv — NLP / Computation & Language research 2mo ago NeuPAT: Neuron-aware Plasticity Allocation Tuning for Language-Preserving MLLMs arXiv:2608.08107v1 Announce Type: new Abstract: Multimodal expansion of large language models (LLMs) enables new perceptual capabilities but often compromises the language intelligence acquired during pretraining. In this work, we investigate this phenomenon from the perspective… 8 arXiv — NLP / Computation & Language research 2mo ago Tied Trit-Planes: Constraining PTQTP to a Uniform Nine-Level Quantizer, with a Persistent Folded Format for Disk-Streamed Mixture-of-Experts Serving arXiv:2608.08910v1 Announce Type: new Abstract: PTQTP decomposes LLM weight matrices into two ternary (trit) planes with two free per-group scales. Tying the scales to a fixed ratio of three collapses the decomposition into a single uniform nine-level quantizer, a known… 29 r/LocalLLaMA community 2mo ago Observations on Muse-Glimmer reasoning traces being noticeably different from qwen / gemma models and questions for you guys Just downloaded the model, UD-Q5_K_XL quant, asked it to generate a long story to test out reasoning and speed with dflash (super fast btw, ~ 90 to 160 tok/s on a 5090 depending on task) and was surprised at the reasoning traces, they are so unlike anything i've seen recently… 5 Hugging Face Daily Papers research 2mo ago Capek 0.5: An Execution-Centric Vision-Language Model for Embodied Intelligence Abstract Vision-language models are increasingly serving as the reasoning core of embodied agents. Robot execution is inherently iterative: each action reshapes the scene and physical state, continually renewing what must be perceived, reasoned about, and verified. Meeting these… 25 arXiv — Machine Learning research 2mo ago CrystalGRPO: Target-Aligned and Coverage-Preserving Reinforcement Learning for Flow-Based Crystal Structure Prediction arXiv:2608.06582v1 Announce Type: new Abstract: Flow-based generative models can efficiently produce candidate structures for crystal structure prediction (CSP), but their pretrained objectives do not directly optimize downstream target recovery. Reinforcement-learning… 33 arXiv — Machine Learning research 2mo ago Multi-Level Modeling of Large Language Model Inference Latency and Energy via Hybrid Analytical--Machine-Learning Predictors arXiv:2608.06723v1 Announce Type: new Abstract: The rapid scaling of Large Language Models (LLMs) has significantly increased computational cost, energy consumption, and inference latency, making accurate estimation essential for sustainable artificial intelligence deployment… 8 arXiv — Machine Learning research 2mo ago CubicQuant: Parametric Non-Uniform Codebooks for High-Throughput LLM Inference with 1-8-Bit Weights arXiv:2608.06763v1 Announce Type: new Abstract: Weight quantization for large-language-model inference must balance adaptive reconstruction levels with representations regular enough for efficient GPU execution. Uniform integers constrain each group to a linear grid. Low-bit… 9 arXiv — Machine Learning research 2mo ago MAUPITI: On-Device Prototype-Based Learning on a Smart Infrared Sensor arXiv:2608.07192v1 Announce Type: new Abstract: Low-resolution infrared (IR) array sensors represent an interesting solution for privacy-preserving human sensing in embedded systems. In this letter, we describe a smart multi-pixel IR sensor integrating a 16$\times$16 thermal… 13 arXiv — Machine Learning research 2mo ago TOFD: Target-Oriented Feature Decoupling against Poisoning Attacks in Split Federated Learning arXiv:2608.07274v1 Announce Type: new Abstract: Split Federated Learning (SFL) facilitates privacy-preserving collaborative training with reduced client-side overhead. However, its split architecture introduces unique attack surfaces, rendering it vulnerable to diverse poisoning… 22 arXiv — Machine Learning research 2mo ago Residual Algebra for Representation-Preserving Learning arXiv:2608.07349v1 Announce Type: new Abstract: Learning from heterogeneous representations is usually reduced to feature concatenation, which erases which representation produced an error. We instead algebraize the residual: a representation is a typed object that owns both a… 14 arXiv — Machine Learning research 2mo ago Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving arXiv:2608.06557v1 Announce Type: cross Abstract: The reasoning and agentic capabilities of large language models have expanded the range of applications they support, from short interactive exchanges to long, compute-heavy requests. LLM serving platforms today define… 34 arXiv — NLP / Computation & Language research 2mo ago Model Confidence Under Answer-Preserving Attacks: An Informativeness-Manipulability Frontier arXiv:2608.06571v1 Announce Type: cross Abstract: Deployed vision-language systems often gate their answers on confidence, making confidence robustness relevant to oversight. We study confidence readouts under white-box, image-only attacks constrained to preserve the generated… 11 r/LocalLLaMA community 2mo ago [2606.05682] Beyond Output Matching: Preserving Internal Geometry in NVFP4 LLM Distillation Demand for low-precision inference, including NVFP4-based approaches, has grown as large language models are increasingly deployed in latency and cost constrained production environments. Quantization-aware distillation (QAD) helps recover accuracy lost under low bit… 32 r/LocalLLaMA community 2mo ago Two flags took the official Ling-3.0-flash INT4 from 20.8 to 38.7 tok/s on one DGX Spark The official INT4 does load on a single DGX Spark. The naive config just leaves most of its speed on the floor, 20.8 tok/s. Two changes take it to 38.7. Quick context on where this comes from: I work on Ling at inclusionAI, and none of these numbers are mine. sudoingX on X ran… 27 r/LocalLLaMA community 2mo ago Memory Bandwidth problems with Intel Sapphire Rapids I have a Xeon w7-3465 and 4 sticks of RDIMM DDR5-4800 with a theoretical max bandwidth of 153GB/s. I am trying to run DeepSeek-V4-Flash-0731 as it is an MoE and the weights are in MXFP4, so I should roughly process 150GB/6.5GB tokens per second as it has 13B Experts. The problem… 11 r/LocalLLaMA community 2mo ago enabling PCI-E p2p for consumer Nvidia cards will yield you more than you think Disclaimer - no LLM was used to write this post/note As larger post about my setup will come later, want to give heads-up to folks who use VLLM and >= 2 GPUs. So I have pretty meaty server (8 channel AMD EPYC, ~150GB/s RAM bw) and 4x5060Ti 16GB in PCI-E 4.0 8x mode. Considering… 29 r/LocalLLaMA community 2mo ago Building a zero-dependency C inference engine for BitNet (1.58-bit) - lessons from hitting 36 tok/s on a Xeon CPU Over the past few months I have been building a CPU-first inference engine from scratch in pure C99 (no Python, no CUDA, no BLAS, just GCC and make). The focus has been running 1.58-bit ternary models natively without heavy runtime overhead. Currently it hits 36.25 tok/s on… 30 r/LocalLLaMA community 2mo ago Qwen3.6 27B + 35B on vLLM, single R9700 (gfx1201) I've been tuning my new Radeon AI Pro R9700, and figured that this would be useful information for people who are trying to optimise their setups. I'm pretty happy with these results and looking forward to Qwen3.8.. Summary below provided by Claude (which helped me configure it… 18 r/LocalLLaMA community 2mo ago Qwen 35B-A3B MoE vs 27B dense in local coding tests: ~4× faster, much smaller quality gap than I expected I compared Qwen 35B-A3B MoE against Qwen 27B dense on a series of local coding-maintenance tasks. On my R9700/llama.cpp setup, the MoE model generated about 3.9× faster (~116 vs ~30 tok/s) , but the coding-quality difference was much smaller than I expected. Both usually handled… 8 r/LocalLLaMA community 2mo ago Serving Deepseek v4 Flash 0731 on 2x DGX Spark — 5-7 GB OS headroom, what would you do to lower VRAM usage and increase OS available RAM? Hey all, I'm serving DSv4Flash 0731 on a cluster of 2x DGX Sparks but am running into constant issues with having almost no RAM (unified memory) left for the OS/cache and I'd love to hear the community feedback on what I could do to get more RAM for headroom. The DGX has an… 32 r/LocalLLaMA community 2mo ago DeepSeek v4 Flash 0731 on H100 node Hi, I've been playing with recent DeepSeek Flash updated model and it seems excellent. Like talking to Claude 4.8. I'm currently deploying it on VLLm and I'm in progres of tuning performance. I seem to hit the point where I trade decode with prefill by juggling batching knobs.… 21 r/LocalLLaMA community 2mo ago A llama.cpp PR makes Q2_0 3.0–3.6x faster on x86 CPUs, 8B decode goes 2.39 → 8.20 tok/s I was going through the current llama.cpp CPU PRs and #26348 stood out because this isn't the usual +5% kernel optimization. It adds an x86 VNNI implementation for the Q2_0 × Q8_0 dot product, and the author's controlled CPU-only benchmarks show roughly 3–3.6x higher throughput… 24 r/LocalLLaMA community 2mo ago Anyone running DeepSeek-V4-Flash-0731 on MI325X with vLLM? Mine is behaving completely broken Is anyone here successfully running DeepSeek-V4-Flash-0731 locally with vLLM , especially on AMD MI325X? My setup: GPU: 1x AMD Instinct MI325X Model: deepseek-ai/DeepSeek-V4-Flash-0731 vLLM: 0.26.0 ROCm image --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4… 20 Hugging Face Daily Papers research 2mo ago SkillZip: Contract-Preserving Graph Compression for Scalable Agent Skill Libraries Abstract SkillZip compresses reusable procedural skills into contract-preserving, executable graph units to enable efficient retrieval and expansion under limited context budgets. Generated by thinkingmachines/Inkling-Small Large Language Models (LLMs) increasingly act as agents… 14 arXiv — Machine Learning research 2mo ago When Does Consensus Mean Correctness? Measuring the Agreement-Accuracy Coupling with Semantics-Preserving Re-Rendering arXiv:2608.05670v1 Announce Type: new Abstract: A model's agreement across perturbed inputs is used both as a label-free reliability signal and as a self-training target, on the premise that agreement tracks correctness. That coupling is rarely measured directly: natural-image… 8 arXiv — Machine Learning research 2mo ago LLM Inference Under Bursty Workload Distribution: Modifying the WAIT Algorithm arXiv:2608.06135v1 Announce Type: new Abstract: Large Language Models (LLMs) such as ChatGPT and Claude are widely used for information retrieval and problem-solving. Recent work has focused on improving scheduling algorithms to boost throughput while maintaining low latency.… 12 arXiv — Machine Learning research 2mo ago Hypothesis Testing with Conditional Queries: Learnability and the Value of Interaction arXiv:2608.06262v1 Announce Type: new Abstract: Model evaluations may fix all tests before observing any responses or select later tests using earlier responses. We study this choice in a conditional-query model on a finite outcome space $\mathcal{X}$ with $|\mathcal{X}|=N$. We… 17 arXiv — NLP / Computation & Language research 2mo ago Example-Guided Prompting for Document-Level Text Simplification arXiv:2608.05447v1 Announce Type: new Abstract: Document-level text simplification requires large language models (LLMs) to rewrite complex documents while preserving meaning, readability, and discourse coherence. Although prompt-based LLMs have shown promising performance, they… 30 arXiv — NLP / Computation & Language research 2mo ago SkillZip: Contract-Preserving Graph Compression for Scalable Agent Skill Libraries arXiv:2608.05604v1 Announce Type: new Abstract: Large Language Models (LLMs) increasingly act as agents whose procedural knowledge is stored in reusable skill packages and loaded at inference time. As skill libraries grow, a central challenge is to expose the smallest sufficient… 25 r/LocalLLaMA community 2mo ago I ported vLLM's serving stack to C++20: 66 MiB binary, no Python at inference, output checked token-for-token against vLLM I'm the author, so discount the enthusiasm accordingly. This is an unaffiliated community port, not endorsed by the vLLM project, which it uses to verify its correctness. What started it: I love vLLM, but a vLLM install here is 9.1 GiB of virtualenv, and I wanted to embed… 36 r/LocalLLaMA community 2mo ago 2 x 5070ti Qwen 27B full config / stats Following up on yesterday's post about running everyone's faves on 2 x 16gb cards while maximizing performance and KV. Previous post data used abandoned Cu130 VLLM image. Stats here are done on cu129-nightly. Which has the KV cache connector fixes and performance improvements.… 34 r/LocalLLaMA community 2mo ago Auto-fit vs tuned MoE offload: 564 → 1330 pp tok/s, unchanged decode (Qwen3.6-35B-A3B Q6 / RTX 3090) TL;DR: On a Qwen3.6-35B-A3B Q6 setup sized for 64K context on a 24GB RTX 3090, spilling eight MoE expert layers to CPU freed enough VRAM to increase -b from 512 to 1024 and -ub from 128 to 512. Prompt processing improved by 2.36×, while generation speed remained unchanged within… 9 r/LocalLLaMA community 2mo ago Final optimization: from ~10 tok/s to ~15 tok/s on DeepSeek-V4-Flash-0731 at 128K ctx - 1 RTX 3090 J'ai consacré beaucoup de temps à l'optimisation de DeepSeek-V4-Flash-0731 GGUF sur une seule RTX 3090. Mon exigence absolue pour chaque configuration était la suivante : Le modèle doit rester utilisable avec une fenêtre de contexte de 128 000 jetons. J'ai testé les différentes… 4 r/MachineLearning community 2mo ago Observations: non-instructional text prefix may bypass RLHF constraints without adversarial prompting? [D] I've been running informal experiments on RLHF-aligned LLMs and consistently observing something I can't fully explain. Posting here to get feedback and find out if this is a known phenomenon or if my methodology is flawed. The observation Inserting a long, thematically coherent… 38 arXiv — Machine Learning research 2mo ago Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs arXiv:2608.04048v1 Announce Type: new Abstract: Serving large language models (LLMs) under diverse deployment constraints requires flexible trade-offs between accuracy, memory footprint, and throughput. However, conventional quantization methods typically require a separate… 6 arXiv — Machine Learning research 2mo ago Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms arXiv:2608.04074v1 Announce Type: new Abstract: Long-context LLM decoding reads the key-value (KV) cache at every step. Loading it takes longer than computing attention over it, so throughput is bandwidth-bound. Hence, reducing the cache size can raise both decoding speed and… 13 arXiv — Machine Learning research 2mo ago Transferable Dual-Stream Representations for Mesoscale-Preserving Sea Surface Temperature Downscaling arXiv:2608.04230v1 Announce Type: new Abstract: Deep learning models for scientific spatio-temporal downscaling often minimize reconstruction error while failing to preserve physically meaningful multi-scale structure. For sea surface temperature prediction, this can yield… 24 arXiv — Machine Learning research 2mo ago CheMLFlow: An Open-Source Platform for Cheminformatics and Materials Informatics Applications arXiv:2608.04942v1 Announce Type: new Abstract: CheMLFlow is an open-source platform for building and executing end-to-end, high-throughput, and agentic workflows for scientific and technological applications. CheMLFlow targets a common bottleneck in scientific machine learning… 34 arXiv — Machine Learning research 2mo ago SSTQ:Privacy-Preserving Vector Quantization via Subsampled Stochastic TurboQuant arXiv:2608.05127v1 Announce Type: new Abstract: Achieving local differential privacy in distributed optimization while maintaining low communication cost remains challenging. Existing vector quantization methods, such as vqSGD, use high-dimensional geometric constructions but… 19 r/LocalLLaMA community 2mo ago Inkling-Small 276B-A12B at ~2.9 tok/s on <10gb memory A follow up to the launch of Mference , it now supports and runs Inkling-Small 276B-A12B . Inkling-Small (Thinking Machines, Apache 2.0), from the pipenetwork/Inkling-Small-MLX-4bit conversion: 276B total, ~12B active, 3.4 GB resident set , ~148 GB on disk. Measured on my M5,… 6 r/LocalLLaMA community 2mo ago Ling-3.0-flash MXFP4 released and running locally on one DGX Spark. In tests: ~80 tok/s decoding 2,500–3,500 tok/s long-input prefilling Smooth use by 3–4 concurrent users Private, on-device inference for coding, agents, and offline batch jobs   submitted by   /u/niacolhealth [link]   [comments] 30 r/LocalLLaMA community 2mo ago LFM2.5-2.6B on a OnePlus 13 at 17 tok/s ~ Pure CPU As you all know the model is 2.69B parameters with a 128K context window and purpose-built for multi-step agent workflows. What you are seeing is the Q4_K_M GGUF running on my own inference engine built from scratch. The TUI is my own device probe suite running through ADB… 27 llama.cpp releases dev-tools 2mo ago b10282 server: Adding spec-decode counters to /metrics endpoint ( #26389 ) server: add spec-decode counters to /metrics endpoint server: fixed review comments and now aligned param names exactly with vLLM. Website: https://llama.app macOS/iOS: macOS Apple Silicon (arm64) macOS Apple… 28 arXiv — Machine Learning research 2mo ago Designing a Good Virtual Node: Addressable and Cardinality-Preserving Global Memory for Message Passing Architectures arXiv:2608.02709v1 Announce Type: new Abstract: Virtual nodes give message-passing neural networks a simple global communication route, but the standard node--VN--node pipeline compresses the graph into one homogeneous state and broadcasts it identically to every node. Building… 36 arXiv — Machine Learning research 2mo ago Schedule-Informed Temporal Fusion Forecasting of Hourly Airport Security-Checkpoint Throughput arXiv:2608.02950v1 Announce Type: new Abstract: Checkpoint staffing requires accurate forecasts of when screening demand will occur, yet flight schedules record departure times rather than passenger arrival times at security checkpoints. This study develops a framework that… 18 arXiv — Machine Learning research 2mo ago AS-FedBridge: Pseudo-Spike Bridge Distillation for Heterogeneous ANN-SNN Federated Learning arXiv:2608.03324v1 Announce Type: new Abstract: Federated learning enables collaborative model training across distributed edge devices while strictly preserving data privacy. To facilitate practical deployment on resource-constrained edge devices, Spiking Neural Networks (SNNs)… 22 arXiv — Machine Learning research 2mo ago Pin Once, Swap Light: Subspace-Aligned Centroid-Residual Training for Efficient Ultra-LoRA Serving arXiv:2608.03579v1 Announce Type: new Abstract: Modern multi-tenant Low-Rank Adapters (LoRAs) serving systems concurrently host tens to hundreds of LoRA adapters. Though powerful, this introduces a critical system dilemma between serving efficiency and task performance:… 11 arXiv — NLP / Computation & Language research 2mo ago Single Canonical Prompts Underestimate LLM Safety's Surface-Form Sensitivity arXiv:2608.02665v1 Announce Type: cross Abstract: A benchmark score is a measurement instrument, yet most benchmarks read each item at a single canonical surface form. We ask whether that reading is faithful: when an item's intent is held fixed and only its meaning-preserving… 38 Hugging Face Daily Papers research 2mo ago JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusion Abstract Real-time video editing requires low-latency causal generation with bounded computational resources while preserving source fidelity and long-term temporal consistency. We present JoyAI-Video-Edit, a 16B-parameter autoregressive diffusion framework for real-time,… 32 Page 10 of 10 · 500 articles ← Newer
