News / #inference Tag Inference 500 articles archived under #inference · RSS Sign in to follow Hugging Face Daily Papers research 1mo ago RECAP-Forcing: Retaining Content Appearances for Long Video Generation Abstract RECAP-Forcing improves long video generation by indexing memory according to appearance novelty rather than recency, preserving key-value caches for newly visible content to maintain long-range consistency without extra training. Generated by… 6 r/LocalLLaMA community 1mo ago Question: Why is prefill unbelievably faster in vLLM than other inference engines? I only started using some vLLM forks recently in a 4 x 48GB 4090 system. DS4F - ~5000pp/180tg (DSpark) Qwen3.8 Flash next - ~7500pp/135tg (MTP) This is amazing, like having the API in my house. But it's also really hard to go back. It's weird that we never come close to prefill… 27 Hacker News — AI on Front Page community 1mo ago Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s I built slotstream, a way to run Qwen3.8-Flash-Next 4-bit on a low-memory mac starting from 16GB, a 125B parameter model that would need 100GB+ memory/RAM, thanks to expert-offloading/ssd-streaming. Easy to install/update, and mac-native using MLX and Swift. It ships with… 34 Hugging Face Daily Papers research 1mo ago Keep-or-Drop? Adaptive Tokenizer for Compact Video Representation Abstract KATok is an adaptive transformer-based video tokenizer that selectively drops uninformative tokens to achieve data-dependent compression while preserving spatial consistency for diffusion-based generation. Generated by thinkingmachines/Inkling-Small Latent diffusion… 10 arXiv — Machine Learning research 1mo ago MEL: Coordinate-Preserving EEG Tokenization for fMRI Translation arXiv:2608.29304v1 Announce Type: new Abstract: Translating electroencephalography (EEG) into functional magnetic resonance imaging (fMRI) is important for medical neuroimaging, clinical brain-state monitoring, and multimodal neural decoding, because it aims to infer spatially… 34 arXiv — Machine Learning research 1mo ago Creation begins with understanding: LLMs as strategy designers for privacy-preserving tabular data synthesis arXiv:2608.29674v1 Announce Type: new Abstract: Sharing tabular data in high-stakes domains is constrained by privacy regulations. Synthetic data offer a promising alternative, but deep generative models are costly to train and difficult to audit, while LLM-based methods often… 28 arXiv — Machine Learning research 1mo ago INTERVenE: Temporal-Abstraction-Interval Based Transformers for Short-Horizon Medical Event Prediction arXiv:2608.29901v1 Announce Type: new Abstract: Electronic Health Record (EHR) prediction models in the intensive care unit must learn from sparse and irregular measurements while preserving the clinical meaning of time and supporting transparent decision-making. We present… 21 arXiv — NLP / Computation & Language research 1mo ago When to Adapt: Conditional Memory Adapters for Retention-Preserving Domain Specialization arXiv:2608.29327v1 Announce Type: new Abstract: Large language models deployed in specialized domains must improve in-domain performance without sacrificing general capabilities. Existing parameter-efficient fine-tuning methods are typically always on: their learned… 38 r/LocalLLaMA community 1mo ago Qwen3.8-Flash-Next in llama.cpp from CPU-only to 96GB VRAM: 8.5 to 109 tok/s, max context and parameters test. My findings on RTX 6000 PRO. Hey guys, I tested Qwen3.8 Flash with llama.cpp from CPU-only to the full 96GB of my RTX PRO 6000. Short version: CPU-only reached 8.34 tok/s at a 2K prompt Full 96GB reached 109.07 tok/s At 245K context, 24GB to 96GB gave 14.89 to 21.61 tok/s The 96GB advantage over 24GB… 6 Hugging Face Daily Papers research 1mo ago GGSS: Geodesic-Gated Spherical Steering for Inference-Time Debiasing of Generative Vision-Language Models Abstract GGSS reduces demographic bias in generative vision-language models by steering visual tokens along geodesic arcs with an adaptive gate, preserving visual-language accuracy. Generated by thinkingmachines/Inkling-Small Generative vision-language models (VLMs) are… 19 OpenAI official-blog 1mo ago OpenAI supports California’s bill to advance youth AI safety OpenAI supports California SB 1119, advancing strong, age-appropriate AI safeguards for teens while preserving opportunities to learn, create, and explore. 28 arXiv — Machine Learning research 1mo ago The Approximation Rank of Softmax Attention: Sharp Geometric Laws and Robust Interaction Dimension arXiv:2608.28150v1 Announce Type: new Abstract: Which geometry controls the rank complexity of normalized softmax attention? We study maximum-row-$\ell_1$ approximation rank, exactly the least unrestricted rank preserving every bounded vector-valued output. Two sharp worst-case… 19 arXiv — Machine Learning research 1mo ago Performative Privacy: When Differential Privacy Maximizes Utility arXiv:2608.28198v1 Announce Type: new Abstract: Privacy-preserving learning is often motivated by the idea that protecting users' data can preserve trust and thus participation, improving utility in the long term. However, this claim has not been formalized so far. In parallel,… 21 arXiv — Machine Learning research 1mo ago Not to Break, but to Attest: Adversarial Probes for Privacy-Preserving LLM Verification arXiv:2608.27954v1 Announce Type: cross Abstract: Post-deployment changes to large language models can alter behavior while leaving routine outputs largely unchanged, creating a challenge for AI governance when model weights are proprietary. We present a privacy-preserving… 20 arXiv — Machine Learning research 1mo ago Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms arXiv:2608.28044v1 Announce Type: cross Abstract: Large language model (LLM) inference serving is priced by tokens, but GPU energy is consumed over inference windows. This accounting mismatch makes token-normalized metrics incomplete, since average output-token energy can… 9 arXiv — NLP / Computation & Language research 1mo ago Trajectory-Level Speculative Decoding for Diffusion Language Models arXiv:2608.27514v1 Announce Type: new Abstract: Diffusion-based language models (dLLMs) enable parallel token generation through iterative denoising, but existing decoding strategies collapse to single-token generation under low confidence, severely limiting throughput. Unlike… 21 arXiv — NLP / Computation & Language research 1mo ago SimpCue: Cue-Based Prompting for Multilingual Text Simplification arXiv:2608.28042v1 Announce Type: new Abstract: Text simplification aims to make complex texts easier to understand while preserving their original meaning. Recent large language models can perform simplification through prompting, but it remains unclear whether adding explicit… 25 arXiv — NLP / Computation & Language research 1mo ago A Probabilistic Interpretation of KV Cache Eviction arXiv:2608.28293v1 Announce Type: new Abstract: The premise and promise of KV (cache) eviction is simple: higher throughput can be achieved by evicting some entries from the KV cache, at a negligible cost to quality. This holds empirically for many existing methods, though most… 37 arXiv — NLP / Computation & Language research 1mo ago ContextPilot: Teaching Agents for Proactive Context Management via Fine-grained RL arXiv:2608.28476v1 Announce Type: new Abstract: Long-horizon agentic tasks require large language models (LLMs) to iteratively retrieve, integrate, and maintain dispersed information across multi-turn interactions, but preserving all interaction histories leads to a continuously… 27 arXiv — NLP / Computation & Language research 1mo ago Semantic Watermarking with Order-Robust Detection over Sub-sentence Units arXiv:2608.27666v1 Announce Type: cross Abstract: Semantic watermarks tie the mark to sentence meaning rather than token choices, promising robustness to content-preserving edits. However, the detector only observes attacker-supplied text, which can be reworded, reordered, or… 34 Hugging Face Daily Papers research 1mo ago LMSM: LLM Security Framework Inspired by Linux Security Modules Abstract LMSM applies a modular security framework to LLM serving by separating evidence calibration, policy evaluation, and output gating, enabling flexible interpretability-based enforcement without rebuilding request handling. Generated by thinkingmachines/Inkling-Small Large… 19 r/LocalLLaMA community 1mo ago R9V: A designer set of kernels I've been working on for R9700s/RDNA4. Qwen3.8-Flash-Next Unsloth IQ4_XS (w/ TP on 2 R9700s, MTP, SSD n-gram, 128k ctx, vision): TG256 of *78 tok/s* (~3x increase), PP8192 of *1510 tok/s* (~30x increase). TL;DR: Ninfer/DS4 but for RDNA4 Highly custom kernels built for RDNA4, applied to vLLM-Radiance to greatly improve Qwen3.8 Flash Next speeds. This is mostly for dual R9700s with preferably 48GB of RAM or higher, but feel free to tinker. SOTA-Scan/DeepGit report in repo. For… 21 r/LocalLLaMA community 1mo ago Qwen 3.8 Flash Next locally on simple mobile phone at 3.5 tok/s Qwen 3.8 Flash Next (80gb) now at 3.5 tok/s on 12gb mid range android phone thanks to some optimizations and with a low quantization on dense part. I don't want to promote the project, but simply show that it's possible on a $400–$500 phone   submitted by   /u/dai_app… 9 r/LocalLLaMA community 1mo ago Qwen3.8-Flash-Next turns 4xR9700 into a local AI powerhouse! 120 t/s TG and 12k t/s PP single request with optimized vLLM If you own 4xR9700 and were waiting for the model to make them shine, then I have some good news for you! It's running at 80-120 tokens/second for generation and 12k token/second prefill for a single request, using tcclaviger's MXFP4-FP8 quant and custom vLLM image… 14 r/LocalLLaMA community 1mo ago Don't Sleep on EXL3 Quants I'm running Muse Glimmer 30B EXL3-SC 3.00bpw H4, fully resident on my 12GB VRAM GPU at 100K context with Q8\_O KV cache. It's a joy to use a dense 30B model at this size and still get \~30 tok/s on a VRAM-constrained laptop. It's supposed to be only slightly worse than the… 37 r/LocalLLaMA community 1mo ago {INTRESTING PAPER BASED ON HBF}2607.10186] FlashAccel: Leveraging High-Bandwidth Flash (HBF) for High-Throughput LLM Inference HBF gives 8x - 16x more capacity than HBM at same cost, and with bandwidth till 3 tb/s.   submitted by   /u/9r4n4y [link]   [comments] 8 r/LocalLLaMA community 1mo ago Qwen3.8-Flash-Next at 170K context on a single 96 GB card. ~110 tok/s. I used the quantized n-gram to INT4, it's 32 GB, memory-mapped from disk. I confirmed that it works great on 150-160k context, and i was watching all the time my VRAM usage while doing single thread long horizon things - the available VRAM should be enough to push it to over… 12 r/LocalLLaMA community 1mo ago I’ve pushed llama.cpp pretty far for Qwen3.8-Flash-Next — is there any reason not to move to vLLM for 200K+ context? I'm currently running Qwen3.8-Flash-Next on a CMP 170HX 64GB + RTX 3090 24GB, with 80GB system RAM. With llama.cpp I've already spent quite a bit of time tuning it: layer split across the two GPUs, PLE on CPU, q8 KV, Flash Attention, detached MTP draft on the 170HX, and some… 29 r/LocalLLaMA community 1mo ago Qwen 3.8 27B at 50 tok/s with 100k Context on a 16GB GPU! (beellama.cpp) I wanted to share my successful setup for running a Qwen 3.8 27B model with a massive context window on a consumer 16GB GPU (RTX 4070 Ti SUPER). The goal was to fit everything into VRAM without sacrificing quality or speed. 🧠 Key Components Model:… 38 r/LocalLLaMA community 1mo ago use llms to auto annotation your dataset locally hi i make tool for this called llmog it's purpose to make llms free to - auto annotation datasets - reclassification existing yolo datasets running totally local using llama cpp or vllm or use external api you'd rather click than code. 🔗 GitHub: mohamed-em2m/llmog: framework… 36 r/LocalLLaMA community 1mo ago Today I hit 181 toks/s (aggregate) on Qwen3.8-Flash-Next on 2x DGX Sparks Hey all, and hello fellow DGX Spark-ers! Today I managed some pretty crazy numbers: 181 tok/s aggregate on 2× DGX Spark on Qwen3.8-Flash-Next at 512Kcontext (2.8M kvc) I hit 181 tok/s aggregate today across a multi-agent fleet on a 2-node DGX Spark cluster. Single-stream decode… 22 NVIDIA Developer Blog official-blog 1mo ago Deploy an Open Model from Checkpoint to Inference in Two Commands with NVIDIA TensorRT Model Connect Open AI models are evolving faster than ever, but bringing them into native applications can still require model-specific conversion, preprocessing,... 22 Hugging Face Daily Papers research 1mo ago Thinking on Shots: Consistent Multi-Shot Video Editing with Agentic Reasoning Abstract An agentic framework combining LLMs and VLMs enables consistent, multi-instruction editing of long multi-shot videos while preserving spatiotemporal structure. Generated by thinkingmachines/Inkling-Small While generative AI has significantly advanced video editing,… 16 arXiv — Machine Learning research 1mo ago Predicting Quantifiability from Primary Screens to Prioritize Dose-Response Profiling arXiv:2608.26538v1 Announce Type: new Abstract: High-throughput drug screening relies on low-cost primary assays to prioritize compounds for more expensive dose-response profiling, where potency is ultimately quantified. Current screening strategies largely focus on identifying… 20 arXiv — NLP / Computation & Language research 1mo ago Knowledge-Verified Emergent Deception in LLM Agents Under Conflicting Incentives arXiv:2608.26372v1 Announce Type: new Abstract: Large language models are increasingly deployed as autonomous agents serving users on behalf of companies, placing them in settings where user and deployer interests can conflict. When an agent knows that a user is owed something… 23 arXiv — NLP / Computation & Language research 1mo ago Preserving General Capabilities during Domain Specialization with Uncertainty-Calibrated MOPD arXiv:2608.26735v1 Announce Type: new Abstract: Specializing large language models to vertical domains improves domain-specific behavior but often degrades general capabilities such as reasoning, coding, instruction following, and creative writing. We study this domain--general… 29 r/LocalLLaMA community 1mo ago Ninfer and a 5090 with 3.8 27B is making me cry tears of joy it's so good. Built the latest and I'm getting as much as 220 tokens per second and averaging in the 170s, I can't get over it. If anyone on here is on that project, fuckkkin' chapeau man, really incredible job. I can't believe I was able to like double or more my throughput from llama.cpp… 34 r/LocalLLaMA community 1mo ago GLM-5.3-Flash @ DGX Station GB300: ~206 tok/s (single stream), 1M context Hey all! I'm finally doing some cool stuff with my "thinking heater" (h/t u/-TV-Stand- ). I'm still experimenting with GLM-5.2 (in anticipation of 5.3 coming tomorrow, I hope!) and things are very cool so far. With the release of GLM-5.3-flash, I decided to play with it on the… 25 arXiv — Machine Learning research 1mo ago ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Acceleration arXiv:2608.24938v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) models scale capacity for strong quality while keeping per-token compute bounded through sparse expert activation. Yet low-latency MoE serving is increasingly challenging, because it spans two inference… 9 arXiv — Machine Learning research 1mo ago NVExplain: Explaining Time Series Forecasting with Latent Trajectory Analysis and Structure-Preserving Surrogates arXiv:2608.25080v1 Announce Type: new Abstract: Time series forecasting models are widely used in high-stakes settings, yet their predictions remain difficult to interpret because existing post-hoc methods often ignore temporal dependence and fail to provide horizon-specific… 20 arXiv — Machine Learning research 1mo ago Transforms for LLM Quantization: The Great Inversion and Format Co-Design arXiv:2608.25188v1 Announce Type: new Abstract: Most competitive 4-bit LLM research pipelines now open the same way: apply a linear, function-preserving transform (rotation, scaling, permutation, non-orthogonal affine) so the outlier mass sits more favorably against the group… 12 arXiv — Machine Learning research 1mo ago Cooperative Multi-Agent Reinforcement Learning for Adaptive Aggregation in Semi-Supervised Federated Learning with non-IID Data arXiv:2608.25794v1 Announce Type: new Abstract: Federated Learning (FL) enables distributed training of machine learning models while preserving data privacy. However, FL struggles with heterogeneous, non-IID client data distributions, resulting in sub-optimal and biased global… 19 arXiv — Machine Learning research 1mo ago When Pruning Meets Interpretability: Preserving Sparse Autoencoder Robustness in LLMs arXiv:2608.25941v1 Announce Type: new Abstract: Sparse autoencoders (SAEs) are widely used to interpret the internal representations of large language models (LLMs), yet their reliability under post-hoc model compression remains poorly understood. We present a systematic study… 17 arXiv — NLP / Computation & Language research 1mo ago Less can be More: Relieving RAG Bottlenecks via Evidence Frontloading and Pressure-Adaptive Budgeting arXiv:2608.25115v1 Announce Type: new Abstract: Existing methods for improving Retrieval-Augmented Generation (RAG) efficiency mainly optimize downstream LLM generation, such as context compression or serving optimization. However, RAG is an end-to-end system, and its bottleneck… 29 arXiv — NLP / Computation & Language research 1mo ago TOPAS: Workflow-Aware Prefix-State Scheduling for Multi-Agent LLM Serving arXiv:2608.25523v1 Announce Type: new Abstract: Prefix caching introduces a fundamental tradeoff in multi-agent large language model (LLM) serving: retaining a long system-prompt key-value (KV) cache for an agent accelerates future calls, yet it reduces the GPU memory available… 5 r/LocalLLaMA community 1mo ago Qwen3.8 27B C8 at 972 TG / 5,680 PP on 4x MI100 rig ($6.5k) using my new INT8 vLLM fork Yet another vLLM fork thread here, but this time its for older INT8-centric hardware. This is a complete INT8 serving stack for Qwen3.8 27B based on vLLM, AITER, and a 27B GPTQ INT8 quant w/ DFlash2 . Its not just another vibed autoresearch loop. No, vLLM ships with very little… 9 r/LocalLLaMA community 1mo ago Lemonade end-of-summer project update, now serving 15 engines! Hi everyone, it's been a while since I posted so here's an update on what the Lemonade community has been up to this summer. Our overall mission is to enable local AI builders with everything they need to make great apps and agents, while keeping the stack turnkey, portable, and… 23 r/LocalLLaMA community 1mo ago Open Source Kernel in Qwen3.6-35B-A3B for AMD MI350X: 78,498 output tok/s on 8 GPUs So here's the thing, almost everyone use NVIDIA to run their LLMs, we also do the same, a lot of people we've met use like RTX PRO 6000 or even H100, B300 It seems like everyone eyes is looking at NVIDIA. However we do the math that the raw power alone on AMD GPU MI350X is… 9 arXiv — Machine Learning research 1mo ago Calibration-Preserving Pruning: Compression as a Reliability Contract arXiv:2608.23744v1 Announce Type: new Abstract: Split conformal prediction, not the pruning rule, supplies finite-sample marginal coverage once a pruned model is fixed independently of the conformal calibration split. We study the separate efficiency problem: can pruning… 31 arXiv — Machine Learning research 1mo ago PinSieve: Production Selective VLM Serving and a Governed Memory Flywheel for Enterprise Content-Quality Triage arXiv:2608.24040v1 Announce Type: new Abstract: Enterprise AI agents in production often need to be bounded, stateful, observable, and governable rather than fully autonomous. We present PinSieve, a production case study in a large-scale content-quality pipeline. Its deployed… 21 Page 7 of 10 · 500 articles ← Newer Older →
