News / #inference Tag Inference 500 articles archived under #inference · RSS Sign in to follow arXiv — Machine Learning research 1mo ago Conditional GraphGANFed: Optimizing Graph-Structured Molecule Generation in Federated Generative Adversarial Networks arXiv:2608.24610v1 Announce Type: new Abstract: Generative adversarial networks (GANs) have garnered considerable attention in molecular discovery for their ability to generate novel and high-quality molecules. To efficiently train a GAN model while preserving data privacy,… 18 TechCrunch — AI news-outlet 1mo ago OpenAI’s Jalapeño chip is built for fast inference at scale, benchmarks show Tested on SemiAnalysis’ InferenceX benchmark, Jalapeño registered both more tokens per user and more throughput per kilowatt than the currently available state-of-the art. 6 Hugging Face Daily Papers research 1mo ago Beyond the Stability-Exploration Dilemma: Environmental Regularization for LLM Policy Optimization Abstract ERPO replaces action-side policy regularization with input-side query distribution control to stabilize reinforcement learning for language models while preserving response exploration. Generated by thinkingmachines/Inkling-Small Policy optimization (PO) for Large… 4 OpenAI official-blog 1mo ago Jalapeño’s first results show industry-leading speed and efficiency in AI inference Jalapeño is a custom inference chip from OpenAI that delivers faster, more power-efficient AI inference, with higher throughput and lower latency for modern models. 34 Hugging Face Daily Papers research 1mo ago Block3D: Efficient Text-to-3D Generation via Block-Wise Diffusion Abstract Block3D accelerates text-to-3D generation by using block-wise diffusion with confidence-guided correction to reduce inference time while preserving geometric fidelity. Generated by thinkingmachines/Inkling-Small While text-to-3D generation has advanced rapidly,… 13 arXiv — NLP / Computation & Language research 1mo ago Can LLMs Truly Forget? Revealing Unlearning Gaps Through Adversarial Evaluation arXiv:2608.21606v1 Announce Type: new Abstract: Machine unlearning aims to remove the influence of targeted training data from a model while preserving its remaining capabilities, but evaluating whether such information has truly become inaccessible remains challenging. Existing… 19 Hugging Face Daily Papers research 1mo ago TileMix: Tile-Centric Mixed-Precision Attention for LLM Inference Acceleration Abstract TileMix routes attention score tiles to mixed FP16 or INT8 precision within fused dense attention, recovering long-context accuracy while improving prefill throughput without retraining. Generated by thinkingmachines/Inkling-Small Long-context prefill in large language… 5 r/LocalLLaMA community 1mo ago [2608.16157] FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution Source of Claims: https://x.com/Andy_ShuoYang/status/2090856976880472439 Your gaming PC can now serve frontier models at interactive speed using official checkpoints without extreme quantization! Qwen3.6 35B → 8GB RTX 4060 laptop @ 39 tok/s DeepSeek-V4-Flash 284B → RTX 5090… 22 r/LocalLLaMA community 1mo ago Qwen 3.8 27B Aider score I ran the Aider benchmark on Qwen 3.8 27B FP8 with FP8 KV cache 256K context vLLM. The score: 72.9 This matches Gemini 2.5 Pro from 2025-04-12 which also scored 72.9. Beats Claude Opus 4 from 2025-05-25 which scored 72.0. DeepSeek R1 2025-06-06 scored 71.4. It may just be a… 24 arXiv — Machine Learning research 1mo ago FlatLand: Personalized Graph Federated Learning via Tailored Lorentz Space arXiv:2608.21096v1 Announce Type: new Abstract: Federated learning enables privacy-preserving collaborative training, but highly heterogeneous client data remain challenging, especially in graph federated learning where clients possess structurally diverse graphs. Existing… 11 arXiv — Machine Learning research 1mo ago Amplifying the imaging power of digital sky surveys with space telescopes data and generative AI arXiv:2608.20666v1 Announce Type: cross Abstract: While Digital sky surveys provide excellent throughput of image data and can cover a large footprint, their imaging power is normally inferior to that of space-based telescopes. Space-based telescopes, on the other hand, provide… 31 arXiv — NLP / Computation & Language research 1mo ago SAC-Copula: Quality-Preserving Watermarking for Diffusion Language Models via Smooth Correlated Gumbel Fields arXiv:2608.20839v1 Announce Type: new Abstract: Watermarking diffusion language models (DLMs) requires mechanisms compatible with iterative parallel unmasking rather than autoregressive decoding. Existing sampling-based watermarking methods typically inject position-wise i.i.d.… 14 arXiv — NLP / Computation & Language research 1mo ago Quantization-Aware Healing: A Practical Recipe for Recovering Compressed, 4-Bit LLMs arXiv:2608.20953v1 Announce Type: new Abstract: Serving large language models cheaply increasingly means shipping models that are both structurally compressed to a fraction of their parameters and quantized to 4 bits. Together these steps degrade reasoning, mathematics, coding,… 5 arXiv — NLP / Computation & Language research 1mo ago Knowledge-Graph-Gated Defactualization for Style-Controllable and Fact-Preserving Generation in Agentic Conversational AI arXiv:2608.20393v1 Announce Type: new Abstract: Agentic large language models (LLMs) deployed in fact-sensitive applications such as customer support must simultaneously preserve factual correctness and generate responses in a controllable stylistic register. Activation steering… 7 arXiv — NLP / Computation & Language research 1mo ago Target-Aware Calibration Data Selection for Preserving Uncertainty in Quantized Language Models arXiv:2608.21019v1 Announce Type: new Abstract: Quantization is widely used to deploy large language models, but its effect on uncertainty behavior, such as confidence, margins, and abstention, is rarely treated as a primary objective. We frame calibration-data selection for… 29 Hugging Face Daily Papers research 1mo ago CLEAR: Continuous Latent Adapter Routing for Utility-Preserving LLM Safety Alignment Abstract CLEAR uses a hidden-state gate to continuously modulate a safety low-rank adapter, improving LLM safety while preserving utility on benign inputs. Generated by thinkingmachines/Inkling-Small Improving the safety of large language models (LLMs) often comes at the expense… 37 r/LocalLLaMA community 1mo ago Ling Tiny, King of Speed Ling Tiny has now replaced Gemma4-12B in my rig as an auxiliary model doing hindsight operations. This is on a 4060Ti, which is a reasonable GPU available out there, and the speed is phenomenal. Don’t enable MTP, set up the vLLM fork for BailingMoE3. Hope this is useful to… 27 r/LocalLLaMA community 1mo ago I hosted Kimi K3 (2.8T parameters) using 8 B300s. 92 tok/s, $190 per million tokens What I ran: 8x B300 on Modal, $56.79 per hour, vLLM, tensor parallel 8, native MXFP4 Cold boot ~27 min (1.56 TB load, JIT, 51 CUDA graph captures) TTFT 0.92 to 1.02 s, decode 92 tok/s steady, 83 tok/s average over 4 prompts $190 per million output tokens. One clean run is about… 10 r/LocalLLaMA community 1mo ago i finally switched from windows to linux and got a 30-50% boost in speed. This is amazing. All I did was switch from llamacpp on windows to vllm on linux.   submitted by   /u/unraveleverything [link]   [comments] 29 r/LocalLLaMA community 1mo ago Single RTX 5090: Qwen3.8-27B NVFP4 at a real 262K context in vLLM — 77 tok/s short-context, 64.7 tok/s at 128K This is the Qwen3.8-27B setup I actually use every day on one RTX 5090. I wanted to write it down with enough detail that another 5090 owner can reproduce it instead of guessing which memory knobs I used. The short version: the full 262,144-token window fits together with… 25 r/LocalLLaMA community 1mo ago I forked Ninfer 3090 and converted it to run on the CMP170HX - doubled my Qwen3.6-35B from llama.cpp Good afternoon, everyone! I wanted to show the work I've been doing around porting Ninfer over to the CMP170HX ( Github ) So, first, I do want to call out the amazing work that Neroued , Sergiuszm and specifically Don-Chad have all done, to not take away from vLLM, Llama.cpp and… 37 r/LocalLLaMA community 1mo ago Freetokens project is impressive A new project was released yesterday and I have the opportunity to test it today. Papper: https://arxiv.org/abs/2608.16157 Github: https://github.com/FlashML-org/FreeToken My initial tests with the following setup: RTX 5080 (16 GB) DDR6 64GB AMD Ryzen 9 9950X3D I got 100tok/s on… 14 r/MachineLearning community 1mo ago I developed my own quantized LLM from scratch, trained on 30B tokens, deploys in 60 MB [R] I trained a 250M parameter model from scratch on 30B tokens of fineweb. It’s quantized to under 2 bits so the whole deployment is 60 MB and it needs about 80 MB of RAM to run. Runs around 400 tok/s on a normal laptop CPU, no GPU needed. How the long context works: the most… 10 r/LocalLLaMA community 1mo ago I feel like I finally graduated. I finally made the move from LM Studio to vLLM thanks to this post https://www.reddit.com/r/LocalLLaMA/s/NmS9CgHvqz . I may not know what it all means yet but I’m going to start diving into the docs to learn as much as I can. I’m running an endpoint on each of my 3090s one for… 4 r/LocalLLaMA community 1mo ago Ornith-1.5-35B-A3B-NInfer - 250 tok/s, 5-8k prefill, 5090 I tried this model yesterday, and it felt to me like the best one I've tried for a local model for interactive use; the responses and reasoning are very fast, and it actually performs agentic tasks well. The speed is phenomenal. I am running this on Ninfer for Windows -… 37 r/LocalLLaMA community 1mo ago Qwen3.8-27B at 262K context on a Strix Halo + RTX 3090 Ti: 9.5 -> 153 tok/s, and it beats a dual-3090 vLLM box on HumanEval Spent a while treating layer placement, KV format and llama.cpp itself as experimental variables. 159 logged experiments. Numbers first, caveats after. Hardware: AMD Ryzen AI MAX+ 395 (Strix Halo, 128 GB unified) + RTX 3090 Ti on an eGPU link. One llama.cpp process, AMD on… 37 Hugging Face Daily Papers research 1mo ago FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving Abstract FlashPrefill V2 improves long-context serving via mean-corrected sparse attention, optimized GPU operators, and framework integration, achieving large speedups over dense baselines. Generated by thinkingmachines/Inkling-Small Long-context modeling is a pivotal… 9 arXiv — Machine Learning research 1mo ago FleetSieve: Decision-Critical Profiling for SLO-Aware LLM Fleet Configuration arXiv:2608.19659v1 Announce Type: new Abstract: Choosing tensor-parallel (TP) degrees and replica counts for an LLM serving fleet is difficult because performance is not monotonic in TP and the feasible choice can change with load. Exhaustive profiling resolves this uncertainty,… 38 arXiv — Machine Learning research 1mo ago HYDRA: A Heterogeneous Chiplet DSE Framework for Serving Dynamic Hybrid LLM Workloads arXiv:2608.19395v1 Announce Type: cross Abstract: Hybrid Transformer-Mamba large language models (LLMs) enhance long-context efficiency, but their heterogeneous computation and communication patterns complicate efficient hardware acceleration. Chiplet-based architectures offer a… 7 arXiv — Machine Learning research 1mo ago HiRA-CAM: Preserving Fine-Grained Spatial Relevance in Gradient-Based Visual Explanations arXiv:2608.19407v1 Announce Type: cross Abstract: Deep Learning models can include billions of parameters or more, making it difficult to explain their internal transformations and outputs. However, explainability is increasing in importance due to the use of AI in crucial… 17 arXiv — Machine Learning research 1mo ago CacheRoute: Planned Prefix-Affinity Routing for Large-Scale LLM Serving arXiv:2608.19677v1 Announce Type: cross Abstract: Prefix caching avoids prefill only when a repeated request returns to a server that still holds the prefix KV. Cache-blind balancing disperses that reuse; fixed affinity preserves it but can overload a server. CacheRoute resolves… 23 arXiv — NLP / Computation & Language research 1mo ago Transformer Models for Text Summarization: A Comparative Study of BART, BERT, and RoBERTa arXiv:2608.19200v1 Announce Type: new Abstract: Text summarization refers to the task of condensing a document into a shorter version while preserving its key information. Automatic text summarization (ATS), driven by advancements in natural language processing (NLP), has… 10 arXiv — NLP / Computation & Language research 1mo ago Linguistic Holonomy and Statistical Watermarks: Inner Geometry of Meaning-Preserving Transformations arXiv:2608.19369v1 Announce Type: new Abstract: Statistical watermarks for language models live in the freedom of the signifier: they choose among tokens that are nearly equivalent in meaning, and they are therefore eroded by exactly those transformations which move the form of… 31 arXiv — NLP / Computation & Language research 1mo ago FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving arXiv:2608.19758v1 Announce Type: new Abstract: Long-context modeling is a pivotal capability for Large Language Models, yet the quadratic complexity of attention remains a critical bottleneck, particularly during the compute-intensive prefilling phase. Our previous work,… 33 Hugging Face Daily Papers research 1mo ago WithEveryone: Unified Planning and Identity Grounding for Group Image Generation Abstract WithEveryone enables reliable identity-preserving group image generation for up to ten people by grounding identities to layout plans and using region-based identity losses. Generated by thinkingmachines/Inkling-Small Identity-preserving image generation becomes… 17 arXiv — Machine Learning research 1mo ago Entropy-Constrained Adaptive Stochastic Quantization arXiv:2608.18147v1 Announce Type: new Abstract: Adaptive stochastic quantization (ASQ) is a recently introduced quantization approach that optimizes the Mean Squared Error (MSE) for a given input while preserving unbiasedness. It is designed to alleviate the communication and… 14 arXiv — Machine Learning research 1mo ago SIGMA: Symmetry-aware, Intelligent, Geometric, Multi-objective Adaptive Control for Robust, Dependable Traffic Management arXiv:2608.18263v1 Announce Type: new Abstract: Traffic signal control is a complex sequential decision-making problem requiring real-time adaptation and trade-offs among throughput, delay fairness, signal stability, and emergency vehicle priority. Existing RL methods often fix… 34 arXiv — Machine Learning research 1mo ago Geometric Data Perturbation with Noisy-Anchor Alignment for Privacy-Preserving Collaborative Learning arXiv:2608.18749v1 Announce Type: new Abstract: Geometric Data Perturbation (GDP) enables one-shot, privacy-preserving collaborative learning: each participant applies a distance-preserving transformation to its private data and uploads only the resulting representation to a… 21 arXiv — Machine Learning research 1mo ago Enhancing EBSD throughput of battery electrode materials using super-resolution generative adversarial networks arXiv:2608.19117v1 Announce Type: new Abstract: Quantitative microstructural characterization of Li-ion battery electrode materials using electron backscatter diffraction (EBSD) has been proven as a critical method for optimizing cell performance. However, the inherently slow… 13 arXiv — Machine Learning research 1mo ago Bridge Graphical Models: Coupling, Projection, and Current-Preserving Dynamics for Generative Modeling arXiv:2608.19144v1 Announce Type: new Abstract: Continuous-time generative models are often built from endpoint-conditioned bridges, but generation requires a different object: a non-anticipative Markov decoder that only observes the current state and time. We identify this… 16 Hugging Face Daily Papers research 1mo ago FACET: Preserving Source Intent and Executable State in Terminal Task Synthesis Abstract FACET constructs executable terminal tasks by preserving source intent and grounding instructions, solutions, and verifiers in a shared repaired environment to enable scalable agent training. Generated by thinkingmachines/Inkling-Small Training terminal agents requires… 15 r/LocalLLaMA community 1mo ago Reverse-Engineering the RK3588 NPU: Building an Open Compiler to Run GPT-2 at 36 tok/s Last year I posted about hacking the RK3588 NPU to run one vision encoder ( previous post ). This year I opened the whole thing up: reverse-engineered the register format, built an open compiler + runtime, and now GPT-2 and SigLIP run from PyTorch, ONNX, and JAX, no vendor SDK.… 30 r/LocalLLaMA community 1mo ago DFlash2 speeds Qwen 3.8 27B up to 4 times llama.cpp pr #27342 adds dflash2, so i rented an rtx 6000 and ran the same four prompts through four decoding setups on qwen3.8 27B median results over the four tasks: baseline 47.4 tok/s mtp 114.7 tok/s dflash 99.3 tok/s dflash2 140.6. tok/s so on average 3x for dflash2 though… 33 Hugging Face Daily Papers research 1mo ago EDITBRIDGE: Towards Faithful and Efficient Ultra-High-Resolution Image Editing Abstract EditBridge enables efficient ultra high-resolution image editing via a diffusion bridge that translates low-resolution edits to high-resolution outputs while preserving source details through sparse attention. Generated by thinkingmachines/Inkling-Small High-resolution… 11 r/LocalLLaMA community 1mo ago Qwen3.8-27B on 2x 3090 + vLLM + DFlash2: 218 tok/s single request I hacked this together so there's probably more on the table in terms of performance. Measured with the Club-3090 canonical bench suite (bench.sh, 3 warmups + 5 measured runs, temp 0.6 / top_p 0.95 / top_k 20). Prefill: 1342 tok/s @ 10k, 628 tok/s @ 90k Spec-decode: 7 draft… 6 arXiv — Machine Learning research 1mo ago Data-DPO: Direct Preference Optimization for Target Model Data Selection in LLM Post-Training arXiv:2608.16926v1 Announce Type: new Abstract: Data selection in supervised fine-tuning aims to select a small set of effective samples from large-scale candidate data, reducing training cost while preserving model performance. However, existing methods usually treat data value… 15 arXiv — Machine Learning research 1mo ago A Constant-Competitive Algorithm for Dynamic Mixture-of-Experts Serving arXiv:2608.16947v1 Announce Type: cross Abstract: Huang, Lou, and Xiao introduced Dynamic Mixture-of-Experts Serving and gave an O(sqrt(log k))-competitive randomized algorithm for its integral primal problem, where k is the number of replica GPUs beyond the mandatory copy of… 22 arXiv — NLP / Computation & Language research 1mo ago CABLE: Extending the Reach of Memory Retrieval via Complementary Antecedent-Based Linking and Expansion arXiv:2608.17911v1 Announce Type: new Abstract: As LLM agents operate across structured workflows and sessions, preserving long-term history does not ensure that later contexts can recover relevant evidence through a bounded memory interface. We study this evidence-reachability… 7 Hugging Face Daily Papers research 1mo ago FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution Abstract FreeToken is an edge-native Mixture-of-Experts serving system that dynamically maps computation and model state onto heterogeneous local hardware to run large open-weight models on personal machines. Generated by thinkingmachines/Inkling-Small Frontier open-weight… 16 r/LocalLLaMA community 1mo ago Running DeepSeek V4 Flash Q4_K_XL at ~100 tok/s prompt processing on 4× RTX 3060 12GB I managed to run the 143–144 GiB DeepSeek-V4-Flash-0731 UD-Q4_K_XL GGUF on four RTX 3060 12GB cards while keeping a 360k–376k context window. Hardware: CPU: Intel Core i9-10920X, 12C/24T RAM: 128 GB DDR4-3200, quad-channel GPU: 4× NVIDIA RTX 3060 12GB Total VRAM: 48 GB Storage:… 25 Page 8 of 10 · 500 articles ← Newer Older →
