News / #inference Tag Inference 500 articles archived under #inference · RSS Sign in to follow arXiv — Machine Learning research 9d ago HYDRA: A Heterogeneous Chiplet DSE Framework for Serving Dynamic Hybrid LLM Workloads arXiv:2608.19395v1 Announce Type: cross Abstract: Hybrid Transformer-Mamba large language models (LLMs) enhance long-context efficiency, but their heterogeneous computation and communication patterns complicate efficient hardware acceleration. Chiplet-based architectures offer a… 7 arXiv — Machine Learning research 9d ago HiRA-CAM: Preserving Fine-Grained Spatial Relevance in Gradient-Based Visual Explanations arXiv:2608.19407v1 Announce Type: cross Abstract: Deep Learning models can include billions of parameters or more, making it difficult to explain their internal transformations and outputs. However, explainability is increasing in importance due to the use of AI in crucial… 17 arXiv — Machine Learning research 9d ago CacheRoute: Planned Prefix-Affinity Routing for Large-Scale LLM Serving arXiv:2608.19677v1 Announce Type: cross Abstract: Prefix caching avoids prefill only when a repeated request returns to a server that still holds the prefix KV. Cache-blind balancing disperses that reuse; fixed affinity preserves it but can overload a server. CacheRoute resolves… 23 arXiv — NLP / Computation & Language research 9d ago Transformer Models for Text Summarization: A Comparative Study of BART, BERT, and RoBERTa arXiv:2608.19200v1 Announce Type: new Abstract: Text summarization refers to the task of condensing a document into a shorter version while preserving its key information. Automatic text summarization (ATS), driven by advancements in natural language processing (NLP), has… 10 arXiv — NLP / Computation & Language research 9d ago Linguistic Holonomy and Statistical Watermarks: Inner Geometry of Meaning-Preserving Transformations arXiv:2608.19369v1 Announce Type: new Abstract: Statistical watermarks for language models live in the freedom of the signifier: they choose among tokens that are nearly equivalent in meaning, and they are therefore eroded by exactly those transformations which move the form of… 31 arXiv — NLP / Computation & Language research 9d ago FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving arXiv:2608.19758v1 Announce Type: new Abstract: Long-context modeling is a pivotal capability for Large Language Models, yet the quadratic complexity of attention remains a critical bottleneck, particularly during the compute-intensive prefilling phase. Our previous work,… 33 Hugging Face Daily Papers research 9d ago WithEveryone: Unified Planning and Identity Grounding for Group Image Generation Abstract WithEveryone enables reliable identity-preserving group image generation for up to ten people by grounding identities to layout plans and using region-based identity losses. Generated by thinkingmachines/Inkling-Small Identity-preserving image generation becomes… 17 arXiv — Machine Learning research 10d ago Entropy-Constrained Adaptive Stochastic Quantization arXiv:2608.18147v1 Announce Type: new Abstract: Adaptive stochastic quantization (ASQ) is a recently introduced quantization approach that optimizes the Mean Squared Error (MSE) for a given input while preserving unbiasedness. It is designed to alleviate the communication and… 14 arXiv — Machine Learning research 10d ago SIGMA: Symmetry-aware, Intelligent, Geometric, Multi-objective Adaptive Control for Robust, Dependable Traffic Management arXiv:2608.18263v1 Announce Type: new Abstract: Traffic signal control is a complex sequential decision-making problem requiring real-time adaptation and trade-offs among throughput, delay fairness, signal stability, and emergency vehicle priority. Existing RL methods often fix… 34 arXiv — Machine Learning research 10d ago Geometric Data Perturbation with Noisy-Anchor Alignment for Privacy-Preserving Collaborative Learning arXiv:2608.18749v1 Announce Type: new Abstract: Geometric Data Perturbation (GDP) enables one-shot, privacy-preserving collaborative learning: each participant applies a distance-preserving transformation to its private data and uploads only the resulting representation to a… 21 arXiv — Machine Learning research 10d ago Enhancing EBSD throughput of battery electrode materials using super-resolution generative adversarial networks arXiv:2608.19117v1 Announce Type: new Abstract: Quantitative microstructural characterization of Li-ion battery electrode materials using electron backscatter diffraction (EBSD) has been proven as a critical method for optimizing cell performance. However, the inherently slow… 13 arXiv — Machine Learning research 10d ago Bridge Graphical Models: Coupling, Projection, and Current-Preserving Dynamics for Generative Modeling arXiv:2608.19144v1 Announce Type: new Abstract: Continuous-time generative models are often built from endpoint-conditioned bridges, but generation requires a different object: a non-anticipative Markov decoder that only observes the current state and time. We identify this… 16 Hugging Face Daily Papers research 10d ago FACET: Preserving Source Intent and Executable State in Terminal Task Synthesis Abstract FACET constructs executable terminal tasks by preserving source intent and grounding instructions, solutions, and verifiers in a shared repaired environment to enable scalable agent training. Generated by thinkingmachines/Inkling-Small Training terminal agents requires… 15 r/LocalLLaMA community 10d ago Reverse-Engineering the RK3588 NPU: Building an Open Compiler to Run GPT-2 at 36 tok/s Last year I posted about hacking the RK3588 NPU to run one vision encoder ( previous post ). This year I opened the whole thing up: reverse-engineered the register format, built an open compiler + runtime, and now GPT-2 and SigLIP run from PyTorch, ONNX, and JAX, no vendor SDK.… 30 r/LocalLLaMA community 10d ago DFlash2 speeds Qwen 3.8 27B up to 4 times llama.cpp pr #27342 adds dflash2, so i rented an rtx 6000 and ran the same four prompts through four decoding setups on qwen3.8 27B median results over the four tasks: baseline 47.4 tok/s mtp 114.7 tok/s dflash 99.3 tok/s dflash2 140.6. tok/s so on average 3x for dflash2 though… 33 Hugging Face Daily Papers research 11d ago EDITBRIDGE: Towards Faithful and Efficient Ultra-High-Resolution Image Editing Abstract EditBridge enables efficient ultra high-resolution image editing via a diffusion bridge that translates low-resolution edits to high-resolution outputs while preserving source details through sparse attention. Generated by thinkingmachines/Inkling-Small High-resolution… 11 r/LocalLLaMA community 11d ago Qwen3.8-27B on 2x 3090 + vLLM + DFlash2: 218 tok/s single request I hacked this together so there's probably more on the table in terms of performance. Measured with the Club-3090 canonical bench suite (bench.sh, 3 warmups + 5 measured runs, temp 0.6 / top_p 0.95 / top_k 20). Prefill: 1342 tok/s @ 10k, 628 tok/s @ 90k Spec-decode: 7 draft… 6 arXiv — Machine Learning research 11d ago Data-DPO: Direct Preference Optimization for Target Model Data Selection in LLM Post-Training arXiv:2608.16926v1 Announce Type: new Abstract: Data selection in supervised fine-tuning aims to select a small set of effective samples from large-scale candidate data, reducing training cost while preserving model performance. However, existing methods usually treat data value… 15 arXiv — Machine Learning research 11d ago A Constant-Competitive Algorithm for Dynamic Mixture-of-Experts Serving arXiv:2608.16947v1 Announce Type: cross Abstract: Huang, Lou, and Xiao introduced Dynamic Mixture-of-Experts Serving and gave an O(sqrt(log k))-competitive randomized algorithm for its integral primal problem, where k is the number of replica GPUs beyond the mandatory copy of… 22 arXiv — NLP / Computation & Language research 11d ago CABLE: Extending the Reach of Memory Retrieval via Complementary Antecedent-Based Linking and Expansion arXiv:2608.17911v1 Announce Type: new Abstract: As LLM agents operate across structured workflows and sessions, preserving long-term history does not ensure that later contexts can recover relevant evidence through a bounded memory interface. We study this evidence-reachability… 7 Hugging Face Daily Papers research 11d ago FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution Abstract FreeToken is an edge-native Mixture-of-Experts serving system that dynamically maps computation and model state onto heterogeneous local hardware to run large open-weight models on personal machines. Generated by thinkingmachines/Inkling-Small Frontier open-weight… 16 r/LocalLLaMA community 11d ago Running DeepSeek V4 Flash Q4_K_XL at ~100 tok/s prompt processing on 4× RTX 3060 12GB I managed to run the 143–144 GiB DeepSeek-V4-Flash-0731 UD-Q4_K_XL GGUF on four RTX 3060 12GB cards while keeping a 360k–376k context window. Hardware: CPU: Intel Core i9-10920X, 12C/24T RAM: 128 GB DDR4-3200, quad-channel GPU: 4× NVIDIA RTX 3060 12GB Total VRAM: 48 GB Storage:… 25 r/LocalLLaMA community 12d ago Qwen 3.8 27B is faster than expected i ran this model on my two 5060 TI 16GB cards at Q4 in unsloth and LM studio ( i downloaded NVFP4 but didn't try it in vLLM ) i think it runs faster than expected it gives me 50 - 60 t/s with MTP. this is surprising because it's a dense model and Qwen 3.6 was giving me 30t/s… 34 Hugging Face Daily Papers research 12d ago DumpsterCluster: From Dumpster Diving to Serving LLaMA-70B on $60 GPUs Abstract Retired GPUs can form low-cost clusters for LLM inference, but their economic and environmental viability depends heavily on local electricity prices and carbon intensity. Generated by thinkingmachines/Inkling-Small As AI datacenters retire functional GPUs, vast… 9 arXiv — Machine Learning research 12d ago Forward Pass Domain Adaptation (Without Cross-Layer Backpropagation) arXiv:2608.14563v1 Announce Type: new Abstract: Forward-Pass-Only MLP training (FPO) adapts large language models without a backward pass through the model body, achieving 2.7--3.2x the throughput of standard fine-tuning at ~40% less peak training memory, while leaving… 16 arXiv — Machine Learning research 12d ago DumpsterCluster: From Dumpster Diving to Serving LLaMA-70B on $60 GPUs arXiv:2608.14614v1 Announce Type: new Abstract: As AI datacenters retire functional GPUs, vast quantities of still capable accelerators enter secondary markets. This paper investigates whether these retired GPUs can find a productive afterlife to form a DumpsterCluster that can… 16 arXiv — Machine Learning research 12d ago FedImp: Enhancing Federated Learning Convergence with Impurity-Based Weighting arXiv:2608.14654v1 Announce Type: new Abstract: Federated Learning (FL) is a collaborative paradigm that enables multiple devices to train a global model while preserving local data privacy. A major challenge in FL is the non-Independent and Identically Distributed (non-IID)… 30 arXiv — Machine Learning research 12d ago Quantifying Depth Sufficiency in Residual Neural Networks: A First-Order Criterion arXiv:2608.14664v1 Announce Type: new Abstract: How can we determine whether a trained neural network is already deep enough? We study this under a fixed function-preserving residual-growth protocol specifying insertion locations, residual families, zero-output initializations,… 21 arXiv — Machine Learning research 12d ago Probability-Preserving Transformer for the Time-Dependent Schr\"odinger Equation arXiv:2608.15112v1 Announce Type: new Abstract: Solving the time-dependent Schr\"odinger equation (TDSE) via traditional numerical methods is computationally intensive. Transformer models offer a compelling alternative, but standard implementations rely on soft constraints that… 9 arXiv — Machine Learning research 12d ago Spectral Saliency for Machine Unlearning arXiv:2608.15548v1 Announce Type: new Abstract: Machine unlearning (MU) aims to remove the influence of specific training data while preserving model utility. As the name suggests, MU can be viewed as the inverse of learning, using gradient-based updates to reduce the influence… 25 arXiv — NLP / Computation & Language research 12d ago Aborted but Not Forgotten: KV-Cache Retention Breaks Rollback Consistency in Language Agents arXiv:2608.15939v1 Announce Type: new Abstract: Stateful language agents assume a rejected branch can be taken back by clearing it from the application transcript. We show this breaks when the serving session retains key/value (KV) state across the logical abort: the model can… 28 llama.cpp releases dev-tools 13d ago b10456 sycl: fix thread/block count in quantized cpy kernel launches ( #27160 ) Adjusts the thread/block count to be proportional to the size of the quant, reducing under/over subscription. Largest perf improvement is the q4_0 -> f32 path, with, on a Arc 70, throughput goes from 20.21… 10 arXiv — Machine Learning research 13d ago CutClean: Neural Network Pruning for Privacy-Preserving Inference arXiv:2608.13773v1 Announce Type: new Abstract: Neural networks are increasingly deployed in high-stakes applications with growing privacy leakage concerns. We show that this privacy leakage can occur even in the absence of representation imbalances that lead to traditional… 29 arXiv — NLP / Computation & Language research 13d ago QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction arXiv:2608.13966v1 Announce Type: cross Abstract: As large language model inference shifts toward lower precision, post-training quantization (PTQ) becomes increasingly brittle, making quantization-aware training (QAT) essential for preserving model quality. However, QAT… 37 arXiv — Machine Learning research 13d ago Adjacency-Based Spectral Proxy Control of Mobile Communication Agents arXiv:2608.13616v1 Announce Type: cross Abstract: We consider a heterogeneous mobile-agent network composed of uncontrolled task agents and controllable communication agents. The objective is to reposition communication agents online as task agents move. Since throughput-based… 24 arXiv — NLP / Computation & Language research 13d ago A Survey of Large Models in Sports arXiv:2608.14377v1 Announce Type: new Abstract: Sports have witnessed growing global enthusiasm in recent years, serving as a vital force for physical health, cultural exchange, social connection, and economic growth. The rapid advancement of large models, particularly… 24 arXiv — NLP / Computation & Language research 13d ago Kalypso: Relational LLM Serving arXiv:2607.23815v2 Announce Type: replace-cross Abstract: Large language models are increasingly used as semantic operators for filtering, extracting, ranking, joining, and transforming unstructured data. Existing semantic query processing systems invoke request-centric LLM… 29 r/LocalLLaMA community 13d ago Qwen 3.8 2.4T at 288k tokens/s on Nvidia GB300 NVL72 https://developer.nvidia.com/blog/serve-qwen3-8-2-4t-a95b-a-2-4t-parameter-model-with-configurable-reasoning-on-nvidia-gb300-nvl72/ 4k tokens per second per GPU of which there are 72. 350 tokens per second per user "Without additional model tuning, the model achieves a… 13 Hacker News — AI on Front Page community 13d ago Tell HN: Cloudflare silently injects its analytics when you switch nameservers A few hours ago I switched my nameservers to Cloudflare in order to enable R2 bucket serving through my own subdomain, and I found out that it silently had injected a JS analytics snippet in my HTML-only JS-free site textlog.cc — I had to go to the Analytics dashboard, Add the… 22 r/LocalLLaMA community 14d ago If you are at the lowest budget, which you can think of.Which hardware would you recommend to run? qwen 3.8 27b oWith like 50 tokens per second. I currently have a RTX 5070 Ti.   submitted by   /u/InternationalGap3698 [link]   [comments] 17 r/LocalLLaMA community 14d ago Anyone managed to get Qwen 3.8 27B running smoothly on vLLM? Can't get rid of endless thinking Title pretty much says it all. I’ve deployed Qwen 3.8 27B using vLLM on an RTX 6000 Pro (tried multiple vLLM releases and launch recipes), but I can't get it into a usable state because of crazy long reasoning passes. Regardless of the thinking effort setting (xhigh, medium, or… 22 r/LocalLLaMA community 14d ago 5090: Windows or Linux for Qwen3.8.27b I've got a dedicated AI rig sitting here with a RTX 5090 and 96GB RAM and for the past few years have been using Windows 11 and primarily LM Studio, but have also used vLLM, llama.cpp and Ollama. With Qwen3.8.27b I want to get the most out of this model. I get the feeling from… 14 r/LocalLLaMA community 14d ago Deepseek v4 flash Q2 on a single 4090 😅 It freaking worked lol🔥 Deepseek-v4-flash-0731 @UnslothAI 's IXQ2/Q3 checkpoint on one single RTX4090 with just 64 GB of RAM at usable token rate without dspark. All kernels running on Blaze (my custom developed ML compiler + inference engine) - no llama.cpp or vllm in the… 23 r/LocalLLaMA community 15d ago Building an open-source control plane for self-hosted vLLM, what would you want in it? Every time I self-host a model I rebuild the same stuff: start the container, set up a route, check why it died overnight, remember to shut the GPU off before it burns money. So I'm building a panel that handles it. Start/stop models, OpenAI-compatible endpoint, health checks… 25 r/LocalLLaMA community 15d ago Try out this "high" reasoning mode for 27B (tested on VLLM) After a lot of tweaking, I have come to the conclusion that 27B lacks a reasoning mode that is between low and xhigh. The "medium" mode isn't actually medium, it erases the explicit instructions to the model. When medium is enabled, the model acts very differently - to me it… 20 r/LocalLLaMA community 15d ago Ninfer-3090 Hi all, Of course I had to jump on Qwen3.8-27B and get it running on the RTX 3090 port of NInfer . For anyone unfamiliar with it, NInfer is a small, dedicated C++/CUDA inference runtime built around fused kernels rather than a huge general-purpose serving stack. The 3090 fork… 30 r/LocalLLaMA community 15d ago SGLang support for Qwen3.8-27B: 200+ tok/s on 5090, 38 tok/s on DGX Spark (NVFP4 + DSpark) Hey r/LocalLLaMA 👋 This is Kai from SGLang. We just shipped day-0 support for Qwen3.8-27B. To push performance for running this model locally, we combined NVFP4 + DSpark and got: 200+ tok/s decode on a single RTX 5090 and RTX Pro 6000 38 tok/s decode on DGX Spark Here's the… 21 r/LocalLLaMA community 15d ago NInfer day0 support for Qwen3.8 27b: ~200 tok/s generation, with tons of engine improvments Qwen3.8-27B is finally here, and NInfer already has Day-0 support! Weights: https://huggingface.co/neroued/Qwen3.8-27B-NInfer Just update to the latest source and give it a try. On a single RTX 5090, NInfer can still reach around 200 tok/s generation with speculative decoding.… 5 r/LocalLLaMA community 15d ago Are we getting Qwen 3.8 35-A3B? So far, it seems like Qwen 3.8 might drop today as a 27B dense model. If that’s the case, no MoE offloading this time , I used to run Qwen 3.6 35B-A3B at around 70 tok/s on an RTX 3060, but offloading a dense model is a completely different story it can be 100× slower or Even… 29 Hugging Face Daily Papers research 16d ago Knowing When to Quit: Diagnosing and Training LLMs to Abort Futile Reasoning Abstract CaRL uses reinforcement learning with refusal incentives and hindsight augmentation to reduce futile reasoning in large language models while preserving task performance. Generated by thinkingmachines/Inkling-Small Large language models generate computationally… 28 Page 2 of 10 · 500 articles ← Newer Older →
