News / #inference Tag Inference 500 articles archived under #inference · RSS Sign in to follow NVIDIA Developer Blog official-blog 29d ago Dense vs. MoE Models: Active Parameters, Throughput, and When to Choose Each How can a 30B-parameter model activate only 3B parameters per token, and still use the capacity of the larger model? Nemotron 3.5 Lightning illustrates the... 38 r/LocalLLaMA community 29d ago Qwen3.8-27B-NVFP4 1M context. So far so good. I am a beginner, Took a while to get started, get everything right. This setup is native not container. Still not sure if I did this right, or if I can tune this more. Environment=HF_HUB_OFFLINE=1 Environment=VLLM_LOGGING_LEVEL=INFO Environment=VLLM_ALLOW_LONG_MAX_MODEL_LEN=1… 15 r/MachineLearning community 29d ago I trained a 44M parameter quantized LLM from scratch on 45B tokens. It ships in 19.8 MB and runs at ~1,900 tok/s on CPU. [P] Three weeks back , i posted SHADOW-250M here. It got 360 upvotes, 293 on r/LocalLLaMA and 94 GitHub stars. Thank you. That model was 60 MB, ran around 400 tok/s on CPU and could retrieve records from an archive on disk. What it couldn’t do reliably was reason over what it… 26 arXiv — Machine Learning research 29d ago Discovering and Preserving Category Correlation Knowledge via Adaptive Reciprocal Knowledge Distillation arXiv:2609.13199v1 Announce Type: new Abstract: Knowledge distillation aims to improve the performance of lightweight student models by transferring knowledge from larger and more powerful teacher models. However, a substantial size gap between teacher and student models often… 13 arXiv — Machine Learning research 29d ago Data-Efficient Agentic Graph Domain Adaptation via Reliability-Aware Prototype Learning arXiv:2609.14045v1 Announce Type: new Abstract: Agentic learning systems are often required to adapt after deployment by observing new data and reusing prior knowledge under limited supervision or feedback. For graph-structured prediction, Graph Domain Adaptation (GDA) naturally… 36 arXiv — NLP / Computation & Language research 29d ago DenMark: Robust Semantic Watermarking for Diffusion Language Models arXiv:2609.14257v1 Announce Type: new Abstract: Semantic text watermarks encode signals in meaning rather than surface token choices, offering robustness to paraphrasing and other semantic-preserving edits. Existing semantic watermarking methods are primarily designed for… 15 Hugging Face Daily Papers research 29d ago LLaDA-UI: Bringing Block-wise Diffusion to Vision-Language GUI Agents Abstract A 16.7B-parameter mixture-of-experts diffusion vision-language agent achieves strong multimodal GUI performance while preserving block-parallel decoding efficiency. Generated by thinkingmachines/Inkling-Small Diffusion large language models (dLLMs) achieve high decoding… 37 r/LocalLLaMA community 1mo ago Running Qwen3.8-Flash-Next locally on a 12GB VRAM card Now that the dust has settled a bit - here's a write-up on running Qwen3.8-Flash-Next (125B-A6B MoE + 51B n-gram table) on relatively middle-tier hardware (RTX 4070 12GB + 64GB DDR5-5600 + Gen4 NVMe on Linux). I started out with bare 6 tok/s and through latest patches and… 28 r/LocalLLaMA community 1mo ago Intern-S2-397B (multimodal, reasoning, coding, and scientific agent capabilities) Model: https://huggingface.co/internlm/Intern-S2-397B Collection: https://huggingface.co/collections/internlm/intern-s2 From Intern Large Models on 𝕏: https://x.com/intern_lm/status/2099425184587370976 vLLM on 𝕏: Day-0 support for Intern Large Models Intern-S2-397B is now… 17 arXiv — Machine Learning research 1mo ago On-Device Language Models for Privacy-Preserving Stress Prediction: A Multimodal Evaluation on Mobile Health arXiv:2609.11961v1 Announce Type: new Abstract: Stress is a pervasive determinant of mental health and a key target for mobile health interventions. On-device language models (ODLMs) offer privacy-preserving inference without cloud dependency, yet their feasibility for health… 35 arXiv — Machine Learning research 1mo ago Hidden in Rounds: Predicting the Time Cost of 802.11 Contention in Federated Learning arXiv:2609.12903v1 Announce Type: new Abstract: Federated learning over IEEE~802.11 shares the wireless channel among clients that send model updates. We use ns-3 to measure the frame-delivery ratio and saturation throughput for different client densities and offered loads. A… 9 arXiv — NLP / Computation & Language research 1mo ago Chopthin-Consensus Power Sampling: A Diversity-Preserving Approach to LLM Decoding arXiv:2609.12243v1 Announce Type: new Abstract: Inference-time power sampling via Sequential Monte Carlo (SMC) can substantially improve large language model (LLM) reasoning without requiring post-training. However, many existing SMC approaches rely on equal-weight resampling,… 26 arXiv — NLP / Computation & Language research 1mo ago Parameter-Efficient Retrievers for Polish and European Languages arXiv:2609.12913v1 Announce Type: new Abstract: Dense retrieval systems increasingly rely on multi-billion-parameter language models, whose memory and computational requirements make large-scale indexing, frequent corpus updates, and low-latency serving costly. We present a… 12 arXiv — NLP / Computation & Language research 1mo ago Kraken: LLM-based Speech-to-Speech Translation via Low-bitrate VQ and Dual-path Source Conditioning arXiv:2609.13045v1 Announce Type: new Abstract: Speech-to-speech translation (S2ST) has advanced significantly with speech LLMs, offering the potential for joint optimization and preserving non-linguistic information. However, these models struggle with predicting high-bitrate… 27 arXiv — NLP / Computation & Language research 1mo ago From Bench-to-Bedside: A Review of Clinical Trials in Drug Discovery and Development arXiv:2412.09378v4 Announce Type: replace-cross Abstract: Clinical trials bridge basic research and clinical application, serving as essential steps in drug development. This review examines clinical trial phases (Phase I [safety assessment], Phase II [efficacy evaluation],… 38 r/LocalLLaMA community 1mo ago R9V Update: now ~100 tok/s in TG on Qwen3.8 Flash Next IQ4_XS on x2 R9700 + 128GB RAM. Fixed crashes with n-gram SSD streaming, improved diagnostics, plus pinned images. Q4_K_XL now supported, 50 tok/s TG. Pushed out this new update, hopefully decreases the instances of crashes. I torture tested this one for ~12 hours after my fixes and found no instability. Q4 K XL needs more fine tuning, which I will work on in the future. I am simultaneously juggling this + a legitimate… 27 Hacker News — AI on Front Page community 1mo ago Why is Google still serving dodgy ads? Article URL: https://www.atomic14.com/2026/09/13/why-is-google-still-serving-dodgy-ads Comments URL: https://news.ycombinator.com/item?id=49686445 Points: 204 # Comments: 92 38 r/LocalLLaMA community 1mo ago Dear 24G owners, try VLLM you might be able to run Qwen3.8 27B INT4, 144K FP8 KV on RTX 3090 with better speed. (TLDR VLLM AOT) VLLM Benchmark: Prefill, Prompt processing - avg, 871.93 tok/s (3 hours constant running xhigh) - 10K prompt, 1000.26 tok/s (16 runs) - 90K prompt, 743,59 tok/s (16 runs) Decode, tok gen - avg, 38.39 tok/s (3 hours constant running xhigh) - 10K, 42.3 tok/s (16 runs) - 90K, 34… 33 r/LocalLLaMA community 1mo ago I built a serverless hosting platform for LoRA adapters with vLLM It’s always bothered me that after fine-tuning a model for a project, there isn’t a particularly easy way to host it without either running it locally and keeping a GPU on 24/7 or paying for an entire GPU server. There are managed options for LoRA serving on top of vLLM (AWS),… 32 r/LocalLLaMA community 1mo ago Nex-N2.5-mini-MLX-4bit on Apple M5 Max — 133.6 tok/s — llm-bench.io Another new model dropped in the course of this week that is well deployable on consumer hardware: Nex N2.5 Mini I went with the recommended settings for the best generation quality and ran a few benchmarks: temperature: 0.7 top_p: 0.95 top_k: 40 reasoning_effort: high I must… 20 r/LocalLLaMA community 1mo ago 2×RTX 3090 + EPYC box running qwen3.8-flash-next at ~38 tok/s What I have: - CPU: EPYC 7551 (32c/64T, Zen 1) - Board: Supermicro H11SSL-i (SP3), Rev 2.0 - RAM: 128 GB DDR4-2133 (all 8 channels full) - GPU: 2x RTX 3090 (48 GB total, PCIe 3.0) - 1500 W PSU What I run: - Qwen3-Flash-Next (177B total / ~6B active MoE, IQ4_XS) on Ilama.cpp.… 38 r/LocalLLaMA community 1mo ago This draft model is OP on 16 GB cards for Qwen 3.8 27b https://huggingface.co/HermiHg/Qwen3.8-27B-DFlash2-Q2_K_S-MIX-GGUF I used this draft model with https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF with the IQ3_XXS with 128k context and I saw it averaging about 60 tokens per second tg speed on the 16 GB RX 9070 XT. This… 16 Hugging Face Daily Papers research 1mo ago Memory as Plans: World-Action Modeling with Memory-Grounded Planning Abstract MaP-WAM improves non-Markovian robotic manipulation by separating memory-grounded planning from plan-conditioned execution, using compact episodic segment records and progress-calibrated action chunks to maintain fixed inference latency. Generated by… 35 OpenAI official-blog 1mo ago Rapidly scaling online storage to serve over 1 billion ChatGPT users Learn how OpenAI evolved Habitat from a Python library into a globally distributed storage platform serving 1 billion ChatGPT users and 22M requests per second. 4 vLLM releases dev-tools 1mo ago proto-v0.1.0 vllm-proto 0.1.0 16 arXiv — Machine Learning research 1mo ago GEOSTEER: Geodesic Optimization for Activation Steering in Large Language Models arXiv:2609.10658v1 Announce Type: new Abstract: Activation steering provides a lightweight way to control large language models (LLMs) by modifying their hidden activations at inference time. Among these approaches, norm-preserving steering aims to change model behavior without… 24 arXiv — Machine Learning research 1mo ago Phase-Decoupled, Model-Calibrated Power Control for Disaggregated LLM Serving arXiv:2609.11133v1 Announce Type: new Abstract: Datacenter GPU power is the binding constraint on LLM serving capacity, and production serving has shifted to prefill/decode (PD) disaggregation. Deploying NVIDIA's Max-Q inference profile on a disaggregated B200 system, we found… 18 arXiv — NLP / Computation & Language research 1mo ago REVA: Reusable Evidence View Aggregation for Context-Efficient RAG Serving arXiv:2609.11209v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) improves knowledge-intensive large language model (LLM) applications by conditioning generation on retrieved documents, but longer contexts increase latency, key-value (KV) cache memory, and… 38 arXiv — Machine Learning research 1mo ago Optimizing AI Inference Across the Deployment Stack arXiv:2609.10550v1 Announce Type: cross Abstract: AI deployment performance is shaped not by model architecture alone, but by interactions among compression, compiler transformations, and serving policies. Published benchmarks often report latency and throughput under… 22 arXiv — Machine Learning research 1mo ago Adaptive Diffusion Freezing: Privacy-preserving Diffusion Models Against Membership Inference Attacks arXiv:2609.10608v1 Announce Type: cross Abstract: Diffusion models have achieved remarkable success in generative tasks across various areas, however their training process raises significant privacy concerns, particularly under membership inference attacks (MIAs). Prior studies… 11 arXiv — Machine Learning research 1mo ago CARTS: Contextual Autoregressive Rank Transcoding Steganography for Full-Capacity Keyed Text Encoding arXiv:2609.10744v1 Announce Type: cross Abstract: Autoregressive language models can be used to transform a payload text into a stegotext of identical token length by preserving per-position rank information across contexts - a methodology we formalize as Contextual… 11 arXiv — NLP / Computation & Language research 1mo ago Automated Identification of Competing Narratives in Political Discourse on Social Media arXiv:2609.11202v1 Announce Type: new Abstract: Social media platforms have become central to shaping political discourse, serving as arenas where narratives form and evolve, influencing public opinion. Identifying and analyzing these narratives, particularly when they compete… 12 arXiv — NLP / Computation & Language research 1mo ago Cross-Lingual Clinical Annotation Projection as Constrained Text Generation: A Six-Language Study arXiv:2609.11450v1 Announce Type: new Abstract: Background: To determine whether cross-lingual clinical annotation projection can be formulated as a text-preserving, document-level generative task that produces verifiable character-level annotations for multilingual clinical… 8 arXiv — NLP / Computation & Language research 1mo ago LOCUS: Task-Aware Low-Rank Post-Training for Token-Efficient Language Generation arXiv:2609.11739v1 Announce Type: new Abstract: Large language model serving costs scale directly with output sequence length, yet standard preference alignment often inflates response verbosity without improving utility. We study whether the parameterization of post-training… 26 arXiv — NLP / Computation & Language research 1mo ago Recognizing Is Not Reversing: A Controlled Inversion Test of Fact-Preserving News Framing arXiv:2609.11769v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used to analyze and rewrite news, yet current framing studies mainly evaluate generation, detection, or whether rewritten text appears more neutral. They do not directly show whether a… 5 Hugging Face Daily Papers research 1mo ago HyQuant: Hybrid-Precision Quantization for LLM Attention Abstract HyQuant improves low-bit LLM attention quantization by preserving critical vertical-line tokens and local windows in high precision while quantizing the rest, maintaining accuracy with low overhead. Generated by thinkingmachines/Inkling-Small Quantization has been… 38 NVIDIA Developer Blog official-blog 1mo ago How Full-Stack NIM Optimizations Deliver 2.5x More Users on Nemotron 3 Ultra Deploying a large language model is only the first step toward production-ready serving. Production teams also need to serve as many concurrent users as... 6 NVIDIA Developer Blog official-blog 1mo ago High-Throughput Structure Prediction with BioNeMo Inference Runtime Biomolecular structure prediction is now often run at proteome scale, where the goal is to move an entire worklist through the pipeline efficiently. NVIDIA... 4 Hugging Face Daily Papers research 1mo ago Train Smarter, Not Harder: Switching Signal-Guided Training in Active Learning Abstract HybridAL adaptively switches from retraining to fine-tuning during active learning based on online stabilization signals, reducing training time while preserving accuracy and improving calibration. Generated by thinkingmachines/Inkling-Small Training strategy, namely… 6 arXiv — Machine Learning research 1mo ago Robustness of LLM-Generated SystemVerilog Assertions to Semantics-Preserving RTL Transformations arXiv:2609.05658v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly being explored for automating SystemVerilog Assertion (SVA) generation, yet most evaluations report correctness on a single syntactic representation of an input. Such point accuracy… 35 arXiv — Machine Learning research 1mo ago Connecting Score Matching, Maximum Likelihood, and Expectation-Maximization in Mixed Linear Regression arXiv:2609.05688v1 Announce Type: new Abstract: We study variance-preserving diffusion of the response in mixed linear regression (MLR) with unknown mixing weights. Our analysis separates the statistical guarantees of score matching from the loss geometry and optimization signal… 29 arXiv — Machine Learning research 1mo ago LoGIC: Budgeted Context Construction for Node-Level Graph In-Context Learning with Tabular Foundation Models arXiv:2609.05955v1 Announce Type: new Abstract: Tabular foundation models have become powerful graph learners. Systems such as G2T-FM and GraphPFN encode each node as a feature row and make predictions through in-context learning (ICL), with labeled rows serving as the prompt.… 28 arXiv — Machine Learning research 1mo ago FANS: Federated Adaptive Network Search Learning for Heterogeneous Devices arXiv:2609.06106v1 Announce Type: new Abstract: Heterogeneous Federated Learning (HFL) aims to train models across devices with diverse resource budgets while preserving data privacy. Existing HFL methods typically bind training to a small predefined menu of model… 31 arXiv — Machine Learning research 1mo ago When Retain Constraints Conflict: Mitigating Forget-Retain Interference in Tabular Data arXiv:2609.06786v1 Announce Type: new Abstract: Machine unlearning aims to remove the influence of designated training data while preserving model utility, but its behavior on tabular data remains underexplored. This gap is important because tabular prediction is widely used in… 31 arXiv — NLP / Computation & Language research 1mo ago Direct Diversity Optimization for Diverse Successful Trajectories in Preference Post-Training arXiv:2609.10052v1 Announce Type: new Abstract: LLM agents for sequential decision tasks are often post-trained with trajectory-level outcome labels, but such labels provide little supervision for preserving multiple successful branches from the same decision state. We study… 13 arXiv — NLP / Computation & Language research 1mo ago KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpoints arXiv:2609.10266v1 Announce Type: new Abstract: LLM serving systems already reuse KV caches, but only when the reused text sits at the very start of the prompt. Two growing workloads break this condition: a retrieval-augmented generation server assembles a different set of… 29 arXiv — NLP / Computation & Language research 1mo ago IBIB: A Protocol for Measuring Enterprise AI Systems by Serving Route, Not Model Identifier arXiv:2609.10494v1 Announce Type: new Abstract: Enterprises deploy systems, not checkpoints. Usable capability depends jointly on weights, serving route, precision, output contract, and harness, yet all 18 audited benchmarks score advertised model identifiers. We treat this as… 36 arXiv — NLP / Computation & Language research 1mo ago Distribution-Consistent Inference for Dynamic Sparse Mixture-of-Experts arXiv:2609.09241v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) architectures have emerged as a powerful paradigm for scaling model capacity while preserving efficient inference in large foundation models. However, most MoE models use a fixed top-$k$ expert selection… 16 r/LocalLLaMA community 1mo ago Running qwen 3.8 27B iq3 xxs on RTX 3060. Getting anywhere from 10 - 20 tps. Thinking Off . Took about 4 mins and 7 mins. Running on about "IQ3_S - 3.4375 bpw" 37.03.960.932 I slot print_timing: id 0 | task 2665 | prompt processing, n_tokens = 12516, progress = 0.98, t = 37.08 s / 337.55 tokens per second 37.04.777.411… 31 NVIDIA Developer Blog official-blog 1mo ago When to Use Encode-Prefill-Decode Disaggregation to Accelerate Multimodal Model Serving Encode-prefill-decode (EPD) disaggregation is an inference optimization technique for multimodal models that separates the vision encoder stage from the prefill... 27 Page 5 of 10 · 500 articles ← Newer Older →
