News / #gpu Tag Gpu 500 articles archived under #gpu · RSS Sign in to follow arXiv — NLP / Computation & Language research 18d ago Attention-Path Fragility as an Uncertainty Signal in Large Language Models arXiv:2608.11138v1 Announce Type: new Abstract: We propose that a model's uncertainty about a token is reflected not only in the breadth of its output distribution but also in whether a confident prediction is \emph{fragile} under perturbation of its attention pathways. We… 27 arXiv — NLP / Computation & Language research 18d ago Withholding the Completing Chunk: Deterministic Pair-Completion Guardrails for Streaming LLM Output arXiv:2608.10279v1 Announce Type: cross Abstract: Streaming language-model output creates a release-timing problem: complete-response moderation acts after streamed text has escaped, whereas repeated semantic classification of partial text can be costly and unstable. We study a… 21 Hacker News — AI on Front Page community 18d ago Nvidia Nemotron 3.5 Lightning and NeMo Switchyard Article URL: https://blogs.nvidia.com/blog/nemotron-lightning-switchyard-rtx-dgx/ Comments URL: https://news.ycombinator.com/item?id=49263340 Points: 220 # Comments: 113 26 Hugging Face Daily Papers research 18d ago Gaming Without an Attacker: Benchmark Fingerprinting in LLM-Driven Search Under Selection Pressure Abstract Optimized GPU kernel benchmarks reveal that evolutionary LLM proposals exploit evaluation configurations, causing widespread failure to generalize to held-out settings. Generated by thinkingmachines/Inkling-Small Benchmarks for systems that are optimized against the… 11 NVIDIA Developer Blog official-blog 18d ago NVIDIA JetPack 7.2.1 Adds Agentic Video Skills and T3000 Emulation Video is a core data path across NVIDIA Jetson applications, from robotics and intelligent video analytics to industrial automation, healthcare, media... 5 r/LocalLLaMA community 18d ago Muse Glimmer 30B running locally in-browser with custom WebGPU kernels at ~25 tok/s on an M4 Max (same speed as llama.cpp)   submitted by   /u/xenovatech [link]   [comments] 23 r/LocalLLaMA community 18d ago nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16   submitted by   /u/TheRealMasonMac [link]   [comments] 21 Hacker News — AI on Front Page community 19d ago Apple Silicon and macOS VMs: 11–16× Faster LLM Inference with Llama.cpp Article URL: https://github.com/trycua/cua/blob/main/blog/gpu-passthrough-macos-vms.md Comments URL: https://news.ycombinator.com/item?id=49259339 Points: 211 # Comments: 32 26 r/LocalLLaMA community 19d ago nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 · Hugging Face   submitted by   /u/coder543 [link]   [comments] 25 NVIDIA Developer Blog official-blog 19d ago NVIDIA Nemotron 3.5 Lightning Delivers Fast, Accurate Specialized Task Execution for Long-Running Agents Long-running AI agents spend most of their time on high-volume execution: tool calls, result validation, and subagent delegation. Using a frontier reasoning... 34 NVIDIA Developer Blog official-blog 19d ago Route AI Agent Workloads Across Models with NVIDIA NeMo Switchyard Building an AI agent does not end with choosing a single model. Each model has its own strengths, weaknesses, and cost profile, which can shift from one... 4 r/LocalLLaMA community 19d ago Revision Prompting: Trades slow (decoded) output tokens for cheap (prefilled) input tokens. TL;DR: If you re-run the same prompt whenever the input changes, try sending the old input/output plus a diff of the input, and ask the model for a patch to the output. You generate ~2-10x fewer output tokens, and the untouched parts of the output stay byte-identical. This… 22 llama.cpp releases dev-tools 19d ago b10359 ggml-webgpu: fix CI errors from #25025 and #25262 ( #26566 ) test new flash_attn test rebase and fix to disable subgrou matrices when max_kv_tile == 0 delete log output Add i32 support to cpy and enables the all ops test restore the non target ci tests comment out of TODO of… 26 r/LocalLLaMA community 19d ago Tested in Coding: BF16 Muse Glimmer vs BF16 Qwen3.6 27B I'm guessing that many people have been waiting for this comparison. For clarity, both models are running at full FP16 KV-cache. Due to VRAM limitations, Muse Glimmer is running full 262,144 context, whilst Qwen3.6 27B can only run at 147,500 context - full GPU offload in both… 28 Stratechery (Ben Thompson) community 19d ago Nvidia’s Risky Business Nvidia is finding new ways for its customers to raise money, and it's expanding the risk of the AI buildout significantly. 33 r/LocalLLaMA community 19d ago Nvidia reportedly testing lower memory configs of Rubin Ultra as memory shortage bites back — designs tested include as little as 192 GB and step back to HBM4   submitted by   /u/fallingdowndizzyvr [link]   [comments] 16 llama.cpp releases dev-tools 19d ago b10362: tests : disable backend sampler hip multi output (#26878) test-backend-sampler: skip multi_output_sampling_chain on HIP The new multi_output_sampling_chain test uses top_k, whose backend probs path needs CUB (unavailable on HIP), so sampled_probs is null and the test aborts. Add it to the existing HIP skip list alongside the other… 29 arXiv — Machine Learning research 19d ago A Controlled Study of Feature-Based Knowledge Distillation Across Student Designs arXiv:2608.08294v1 Announce Type: new Abstract: Knowledge distillation trains a smaller student to match the outputs of a larger teacher. Feature-based methods also align intermediate representations, but this extra constraint may affect students differently. We study this… 23 arXiv — Machine Learning research 19d ago Gaming Without an Attacker: Benchmark Fingerprinting in LLM-Driven Search Under Selection Pressure arXiv:2608.08722v1 Announce Type: new Abstract: Benchmarks for systems that are optimized against the evaluation signal measure something different from what they claim. We document this concretely in two GPU-kernel-optimization suites with held-out generalization gates:… 24 arXiv — Machine Learning research 19d ago Measuring and Reducing WebGPU Dispatch Overhead for LLM Inference arXiv:2608.08730v1 Announce Type: new Abstract: Large Language Models are deployed to multiple types of environments, from internet browsers to edge devices, and WebGPU serves as a modern cross-platform standard. The engines for browser-based LLM inference have proliferated, yet… 19 arXiv — Machine Learning research 19d ago Beyond Routing: Decoupling Expert Dispatch and Aggregation in Sparse Mixture-of-Experts arXiv:2608.08853v1 Announce Type: new Abstract: Sparse Mixture-of-Experts (MoE) routers commonly use the same scores both to select experts and to weight their already-computed outputs. We study whether these two roles, dispatch and aggregation, should be coupled. On pretrained… 11 arXiv — NLP / Computation & Language research 19d ago APEX-VW: A Document-Level English-Spanish Post-Editing Dataset in the Healthcare Domain arXiv:2608.08059v1 Announce Type: new Abstract: Post-Editing (PE) of Machine Translation (MT) output often involves repeating the same lexical and terminological corrections across many segments, especially in specialised and highly repetitive documents. Despite substantial work… 25 arXiv — NLP / Computation & Language research 19d ago Failure-Aware Long-Form Translation: Design and Implementation of a Recoverable LLM Translation System arXiv:2608.09187v1 Announce Type: new Abstract: A long-form translation request can succeed at the API layer and still produce an unusable result. The output may be empty, truncated, filtered, dominated by source or prompt material, or interrupted after producing text worth… 32 arXiv — NLP / Computation & Language research 19d ago Pragmatic Attack Surface: Vulnerabilities of Implicit Context in Large Language Models arXiv:2608.09551v1 Announce Type: new Abstract: In the era of large language models (LLMs), attackers often manipulate natural language to elicit unsafe or harmful outputs, creating a new natural language attack surface unique to LLM-based systems, where attacks directly exploit… 26 llama.cpp releases dev-tools 19d ago b10355 llama : support multi-output backend sampling ( #25532 ) Enable backend sampling with token speculation Clamp the mask sum before converting it into the sampled index Add a numeric context parameter declaring the maximum outputs one sequence More fixes Don't reuse memory for… 8 llama.cpp releases dev-tools 19d ago b10353 ggml : require contiguous src for ROLL on CUDA and Metal ( #25928 ) ggml_roll only asserts nb[0] == ggml_type_size, so a permuted src is a valid input, but the CUDA and Metal roll kernels index by ne alone and never read the nb strides. A non-contiguous src therefore produced… 25 Hugging Face official-blog 19d ago Build Low-Latency Multilingual Voice Agents: Open Weights & Full Deployment Control with NVIDIA Magpie TTS Back to Articles a]:hidden"> Build Low-Latency Multilingual Voice Agents: Open Weights & Full Deployment Control with NVIDIA Magpie TTS Enterprise + Article Published August 10, 2026 Upvote 4 Maryam Motamedi maryameee nvidia Mikyas Desta mdestanv nvidia Jason Li blisc nvidia… 20 r/LocalLLaMA community 19d ago DeepSeek V4 Flash 0731 is the ‘killer app’ that is going to sell A LOT of DGX Sparks Having a ‘Killer Application’ that everyone wants to use helps sell hardware, plain and simple. DeepSeek V4 Flash 0731 isn’t an app of course, but I think it’s going to be the major catalyst for getting a lot of people to buy a couple of NVIDIA GB10-based systems because: It is… 33 Hacker News — AI on Front Page community 20d ago Humanising LLM Outputs Is Dumb Article URL: https://kuber.studio/blog/Reflections/Humanising-LLM-Outputs-is-Actually-Dumb Comments URL: https://news.ycombinator.com/item?id=49243474 Points: 200 # Comments: 131 5 NVIDIA Developer Blog official-blog 20d ago Run Local Agentic AI Workflows with Meta’s Muse Glimmer on NVIDIA Meta returns to the open source ecosystem with the release of Muse Glimmer, a 30B open-weight dense model with a 120K+ context window built for local AI... 37 Ollama releases dev-tools 20d ago v0.32.7 Muse Glimmer Note: Muse Glimmer is currently available via initial support via Ollama's MLX engine on Apple Silicon. Support for NVIDIA, AMD, and other platforms will be available in the coming days. Muse Glimmer , Meta's newest open model and the first released by Meta… 33 llama.cpp releases dev-tools 20d ago b10336 ggml-webgpu : refactor several wgsl files and simplify flash_attn wgsl. ( #26134 ) Website: https://llama.app macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64… 15 vLLM releases dev-tools 20d ago v0.27.0: [Kimi][MM] disable kimi_vit's dynamic torch.compile for TPU (#51196) Signed-off-by: Linkun Chen github@lkchen.net (cherry picked from commit 7f58e82 ) 11 r/LocalLLaMA community 20d ago Running Qwen 3.5 35B A3B-Q8_0 gguf on a cheap radeon 7600 at 18 token/s I also have 64 gb ddr4 ryzen 5600 Using llama.cpp Ubuntu distro Settings are as follows --n-gpu-layers 999 \ --n-cpu-moe 37 \ --no-mmap \ -ctk q8_0 \ -ctv q8_0 \ -fa 1 \ -c 9000 \   submitted by   /u/Sweaty_Perception655 [link]   [comments] 30 arXiv — Machine Learning research 20d ago The Sparsity Whisperer arXiv:2608.06630v1 Announce Type: new Abstract: Pruning reduces the inference cost of large language models, but existing criteria primarily preserve large activations or reconstruct layer outputs. We argue that this overlooks a key computation performed by particularly… 13 arXiv — Machine Learning research 20d ago CubicQuant: Parametric Non-Uniform Codebooks for High-Throughput LLM Inference with 1-8-Bit Weights arXiv:2608.06763v1 Announce Type: new Abstract: Weight quantization for large-language-model inference must balance adaptive reconstruction levels with representations regular enough for efficient GPU execution. Uniform integers constrain each group to a linear grid. Low-bit… 9 arXiv — Machine Learning research 20d ago Faster Query-Key Learning Sharpens Attention in Self-Attention Models arXiv:2608.06776v1 Announce Type: new Abstract: A standard self-attention layer consists of two interacting circuits: the query-key circuit that governs attention allocation, and the output-value circuit that maps attended representations to predictions. Collapsed and factorized… 8 arXiv — Machine Learning research 20d ago How Molecular Generative Models Organize Molecular Identity arXiv:2608.06956v1 Announce Type: new Abstract: Generative models for matter are often evaluated as samplers over output representations, and their latent spaces are commonly used as proxies for navigating chemical space. Much less is known about how these models internally… 6 arXiv — Machine Learning research 20d ago Stochastic Autoregressive Learning arXiv:2608.07224v1 Announce Type: new Abstract: Motivated by LLMs, which generate outputs by iteratively sampling from next-token distributions, we introduce a PAC-learning model for binary stochastic autoregressive learning. This generalizes the deterministic autoregressive… 13 arXiv — NLP / Computation & Language research 20d ago NTDH: Complex Reasoning for Comprehensive Affective Analysis arXiv:2608.06425v1 Announce Type: new Abstract: Comprehensive affective analysis is challenging for two reasons: it spans heterogeneous prediction tasks with continuous, ordinal, and multi-label outputs, and affective meaning is context-dependent, requiring conflicting cues to… 9 arXiv — Machine Learning research 20d ago Density-Functional Excited-State Gradients and Nonadiabatic Couplings on a Consumer GPU from a Contraction-DAG arXiv:2608.06536v1 Announce Type: cross Abstract: Nonadiabatic dynamics needs an excited-state gradient and an interstate nonadiabatic coupling matrix element (NACME) at every nuclear geometry, and a double-hybrid functional's accuracy has been unavailable for the coupling. We… 7 arXiv — NLP / Computation & Language research 20d ago Beyond "AI Language": The case for the idiolectal nature of LLM output arXiv:2608.06589v1 Announce Type: new Abstract: While large language model outputs are frequently analysed as a collective super variety termed "AI language," this chapter argues that this perspective coexists with distinct, model-specific linguistic signatures akin to human… 10 arXiv — NLP / Computation & Language research 20d ago CreativeInstruct: Scalably Teaching LLMs to Balance Quality, Creativity, and Diversity arXiv:2608.07460v1 Announce Type: new Abstract: While post-training improves the capabilities of large language models (LLMs), it generally lowers their output diversity and creativity, negatively impacting tasks that explicitly require creativity (e.g., story generation) as… 27 r/LocalLLaMA community 20d ago [2606.05682] Beyond Output Matching: Preserving Internal Geometry in NVFP4 LLM Distillation Demand for low-precision inference, including NVFP4-based approaches, has grown as large language models are increasingly deployed in latency and cost constrained production environments. Quantization-aware distillation (QAD) helps recover accuracy lost under low bit… 32 r/LocalLLaMA community 20d ago Pathway's BDH(post-transformer arch) matches GPT2 scaling from 10M to 1B params trained from scratch. runs on Normal GPUs   submitted by   /u/Candid-Tackle-9061 [link]   [comments] 36 r/LocalLLaMA community 20d ago Add a amd 9700 ai pro to a 3x5090 system vs buy a 5070ti for general useage. Hey guys I have this OCD im trying to decide about, I was lucky enough to buy 3 5090 before all the crazy ai stuff started and while that system works fine. Id also bought a razor core egpu that stopped working a little ago and so got sent off for repair, well its come back now… 35 r/LocalLLaMA community 21d ago Underestimated budget solution: radeon 780m iGPU There are so many posts where people complaining about high prices and asking for solution <= 1000 EUR. So, there is one solution to consider: PC/mini PC/laptop on Ryzen 7 260 / Ryzen 9 8945HX /etc CPU with 780m iGPU and 64 Gb of DDR5 RAM. Barebone mini PC costs around 300-400,… 6 r/LocalLLaMA community 21d ago AMD llama.cpp: reducing MTP buffer overhead gave me 64K → 149K context for Qwen 27B Available context length with and without the patch: Model: QWEN 27B ROCm stock patched Vulkan stock patched IQ4_XS Pure, single 16GB GPU 19.456 76.032 68,352 78,592 Q6_K_L on 16GB + 12GB 64,256 149,248 68,864 151,296 The issue is that llama.cpp overestimates the memory needed… 6 r/LocalLLaMA community 21d ago Extremely slow DSpark draft model performance (1-2 t/s) with DeepSeek-V4-Flash on llama-server compared to MTP? Hey everyone, I could use some advice on setting up speculative decoding correctly with llama-server . My Hardware: GPUs: RTX 4090 + RTX 6000 Pro (120GB total VRAM) RAM: 32GB I am currently testing the DeepSeek-V4-Flash-0731 unsloth model using the 137GB Q4 UD-IQ4-NL quant. When… 17 r/LocalLLaMA community 21d ago enabling PCI-E p2p for consumer Nvidia cards will yield you more than you think Disclaimer - no LLM was used to write this post/note As larger post about my setup will come later, want to give heads-up to folks who use VLLM and >= 2 GPUs. So I have pretty meaty server (8 channel AMD EPYC, ~150GB/s RAM bw) and 4x5060Ti 16GB in PCI-E 4.0 8x mode. Considering… 29 Page 7 of 10 · 500 articles ← Newer Older →
