News / #gpu Tag Gpu 500 articles archived under #gpu · RSS Sign in to follow llama.cpp releases dev-tools 22d ago b11109 metal : gate mul_mm_id src1 rescale behind ggml_prec ( #29029 ) metal : gate mul_mm_id src1 rescale behind ggml_prec Assisted-by: Claude Fable 5.1 ggml-webgpu: reject MUL_MAT_ID when src1 precision is F32 cuda/vulkan: reject MUL_MAT_ID in supports_op when src1 prec is F32 fix… 10 NVIDIA Developer Blog official-blog 22d ago Enabling Private High-Performance Production AI Inference with NVIDIA Confidential Computing As large language model (LLM) inference increasingly processes sensitive information and proprietary model context across personal, enterprise, and regulated... 33 r/LocalLLaMA community 22d ago Best value 24/32 GB GPU? Currently sitting on some PC parts that I’m planning to sell off, and i’m considering using the money to upgrade my current gpu (12 gb vram) but if i do so i want to make sure i do my homework. Seen a lot of things thrown around this sub, like a 3090 or 9700, but i honestly… 26 NVIDIA Developer Blog official-blog 22d ago Topology-Aware Workload Scheduling with NVIDIA Topograph AI factories are power-limited systems that deliver maximum value when fully optimized. GPU workload placement is a key optimization. Poor workload placement... 5 TechCrunch — AI news-outlet 22d ago Five AI safety sessions every founder should have on their TechCrunch Disrupt 2026 agenda At TechCrunch Disrupt 2026, five sessions across the AI Stage and Real World AI Stage cover AI safety, featuring leaders from Anthropic, NVIDIA, AWS, Waabi, and more. Register now to save up to $200 before Sept 25. 6 NVIDIA Developer Blog official-blog 22d ago What’s New for Game Developers: DLSS 5 with 3D-Guided Neural Rendering, NVIDIA ACE Updates, and New RTX Kit Capabilities NVIDIA DLSS 5 introduces DLSS 3D-Guided Neural Rendering and granular controls that help game developers add lifelike lighting and material detail while... 25 NVIDIA Developer Blog official-blog 22d ago Accelerating a ROS 2 Node with an AI Agent and NVIDIA Isaac ROS GPU acceleration can speed up compute-intensive robotics workloads, but a fast CUDA kernel alone does not guarantee a fast ROS 2 graph. As messages move between... 28 arXiv — Machine Learning research 23d ago CleanScore: Black-Box Benchmark Audits with Negative Controls and Sensitivity Bounds arXiv:2609.22183v1 Announce Type: new Abstract: Public benchmark scores may reflect skill, prior exposure to the questions, or both, and for most models the training data are unknown. We present CleanScore, a black-box audit using scored outputs only. Each benchmark question… 36 arXiv — Machine Learning research 23d ago Measuring the Checker: Mutation Analysis for GPU-Kernel Benchmark Oracles arXiv:2609.22220v1 Announce Type: new Abstract: Benchmarks for LLM-generated GPU kernels decide correctness with a few random inputs and a loose floating-point tolerance, and their verdicts now feed leaderboards and reinforcement-learning rewards. Recent work agrees these… 38 arXiv — NLP / Computation & Language research 23d ago Evaluating Personal Information Output from Conversational Interactions in Generative AI Systems arXiv:2609.22204v1 Announce Type: new Abstract: This exploratory pilot study evaluates the scope and perceived accuracy of personal information output from ongoing conversational interactions in generative AI systems using GPT-5.2 Instant and GPT-5.2 Thinking, categorized into… 38 arXiv — NLP / Computation & Language research 23d ago Beyond Final-Token Classification: Heterogeneous Readouts for Evidence-Grounded Suicide Risk Detection arXiv:2609.22767v1 Announce Type: new Abstract: The IEEE BigData Cup benchmark combines three prediction problems with different output structures: ordinal suicide-risk classification, multi-label psychosocial factor detection, and extraction of supporting phrases. We introduce… 4 arXiv — NLP / Computation & Language research 23d ago Attributable Post-Rationalization in RAG Citations: A Controlled Reproduction and an RLVR Comparison arXiv:2609.23053v1 Announce Type: new Abstract: A RAG system can hand you the right answer and cite a source it did not actually use. Models output these unfaithful citations via post-rationalization: they write the answer first and then attach a citation to whatever passage… 24 llama.cpp releases dev-tools 23d ago b11090 cuda: fix sm_70 tile compilation error ( #29224 ) The 5-argument load_ldmatrix added in 1884824 only defines tile<16,8>, so the Volta tile<8,4> does not match. See #29222 for details. Building on 1884824 , generalize the tile shape of the 5-argument load_ldmatrix from <16,8> to… 14 r/LocalLLaMA community 23d ago Wow, Mimo 2.6 pro seems to be pretty good, but requires more prompting than Sol Although it makes mistakes and use more tokens, but after some reprompting , It(max) gives really good outputs like on par with 5.6 sol and close to astra High in one task.. IT will likely vary on tasks. I hope ds v4.1pro is gonna be really good or at least as good as Kimi K3.… 27 NVIDIA Developer Blog official-blog 23d ago Simplifying Model Serving Across Multiple GPUs with NVIDIA TensorRT Multi-Device Integration in NVIDIA Dynamo-Triton The compute and memory demands of generative AI increasingly exceed what a single GPU can provide. NVIDIA TensorRT multi-device inference is a new capability... 18 r/LocalLLaMA community 23d ago Huawei shelves global AI chip rollout as China's own demand outstrips supply — AMD and Nvidia no longer have to worry.   submitted by   /u/fallingdowndizzyvr [link]   [comments] 29 r/LocalLLaMA community 23d ago A better coder for the small-GPU/small-RAM crowd! I’ve been working on making small models more capable at agentic coding and work, because most people in the world don’t have the sort of hardware needed to run 3.8-27B, or even 35B-A3B or 9B dense, and I want to extend local agentic coding capability to less privileged users.… 21 r/LocalLLaMA community 23d ago [MASSIVE RELEASE] Supra2-IMG - a tiny 100M text-to-image model - SOTA quality and open release! Hey everyone! It has been quite a while since the last SupraLabs model - but today we've something special for y'all: Supra2-IMG It's a 100M parameter DiT text-to-image model trained entirely from scratch in under 10 hours on a single H100 on Runpod. It can generate… 32 NVIDIA Developer Blog official-blog 23d ago Turn Your Latest Observations Into Timely Weather Decisions With NVIDIA Earth-2 Weather-sensitive industries increasingly have access to observations that offer an earlier, more local view of changing conditions. Energy companies collect... 13 llama.cpp releases dev-tools 23d ago b11071 ci : Upgrade CUDA to 13.4 for Ubuntu CUDA Release Builds ( #29202 ) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48951751 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel… 35 r/LocalLLaMA community 23d ago 16GB (and in many cases 12GB) is the max vram most people will ever reasonably have This sub is, needless to say very niche and skewed towards the high end. There are tons of extremely high end setups here with multiple gpu's etc. Even 24GB is out of reach of most people financially, forget about the 3x3090 or 5090 or even higher setups. Macs/Strix Halo/dgspark… 25 r/LocalLLaMA community 23d ago Deterministic Kittens: fun with Qwen Image 2.1 on an M2 Macbook Pro Having a blast with Qwen Image 2.1 on my M2 Macbook Pro with 32GB RAM, even though it takes 16 minutes to complete the recommended number of iterations with the resolution reduced to 1024x1024. And those are some hot minutes! Above are some entertaining outputs. I strongly… 9 llama.cpp releases dev-tools 23d ago b11069 cuda : tune MMVQ to MMQ crossover for SM70 (Volta) ( #28912 ) tune MMVQ to MMQ crossover for SM70 (Volta) Signed-off-by: Yangyu Chen cyy@cyyself.name Apply suggestion from @JohannesGaessler Apply suggestion from @JohannesGaessler Apply suggestion from @JohannesGaessler… 8 r/LocalLLaMA community 23d ago Gewell - Gemma4 inference engine # the What An engine to run Gemma 4 31B on blackwell under massive concurrency and rather specific workload patterns. I've been waiting for someone to do ninfer but for gemma, and, well, ended up having to do it myself. More models and potentially more gpus are likely to be… 36 llama.cpp releases dev-tools 23d ago b11067 webgpu : add fused gdn + cpy ( #28976 ) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48873164 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:… 20 vLLM releases dev-tools 23d ago v0.30.0: [Build] Fix DeepGEMM CUDA 12.9 release builds (#57554) Signed-off-by: khluu khluu000@gmail.com Co-authored-by: OpenAI Codex noreply@openai.com Co-authored-by: Jee Jee Li pandaleefree@gmail.com (cherry picked from commit eb87980 ) 15 r/LocalLLaMA community 23d ago What's the verdict on Ternary Bonsai 2 27B? Is it worth switching to from Ornith 1.5 9B? Or is there another similarly sized model that's beating both of them? (I head K2 is good but the KV cache is HUGE). Also btw this is for GPU poor folks so please don't suggest some 30B model.   submitted by   /u/PotterSkxawng… 4 r/LocalLLaMA community 23d ago Where are the current GPU VRAM sweet spots? I have been reasonably satisfied with my single R9700 (32GB) as I can run practical quants of Qwen 3.8-27B at good speeds, as well as other similar models in its weight class (Gemma 4 is still my go-to for general knowledge, until I see something better - has that happened?).… 31 arXiv — Machine Learning research 24d ago IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Experts arXiv:2609.21346v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) scales capacity, but existing designs cannot set three quantities independently. For a single token, participation is how many experts contribute knowledge to its output, execution is how many are actually… 17 arXiv — Machine Learning research 24d ago MACE: Memory-Agent Co-Evolution with Adaptive Memory Graphs for Multi-Agent Systems arXiv:2609.21533v1 Announce Type: new Abstract: LLM-based multi-agent systems generate collaboration traces that record how agents plan tasks, verify intermediate results, and repair failures. Reusing these procedures requires preserving an action's prerequisites and the outputs… 15 arXiv — Machine Learning research 24d ago The Weight Is Over - Interactive Diffusion on Consumer GPUs arXiv:2609.21849v1 Announce Type: new Abstract: On-device inference is booming, but the momentum is almost all in language models. Diffusion pipelines are memory hungry, latency-sensitive, and require orchestrating an embedder, a transformer, a decoder, and often further… 13 arXiv — Machine Learning research 24d ago Available Guardrails: Certifying Selective Prediction across ML Systems arXiv:2609.22048v1 Announce Type: new Abstract: A selective predictor acts as a safety gate: it returns an output only when the prediction appears sufficiently trustworthy. Deployments increasingly require this reliability to be certified at a target precision for every… 34 arXiv — NLP / Computation & Language research 24d ago From Task Success to Productive Success: Evaluating Human-AI Collaboration by Quality and Cost arXiv:2609.21117v1 Announce Type: new Abstract: AI productivity is often measured by task completion time, economic value, or improvements in outcome quality. However, these measures usually treat collaboration as a black box where they capture what output was produced, but not… 37 arXiv — NLP / Computation & Language research 24d ago Hallucination-R1: Robustness-Oriented Paraphrase Generation for Factual Consistency arXiv:2609.21227v1 Announce Type: new Abstract: Factual hallucination is commonly defined by incorrect factual outputs. We study a paraphrase-induced hallucination setting, where a model answers a factual question correctly in its original form but generates an incorrect answer… 4 arXiv — NLP / Computation & Language research 24d ago The Hidden Cost of Digits: Number Normalization and WER in ASR Systems arXiv:2609.21084v1 Announce Type: cross Abstract: Modern automatic speech recognition (ASR) systems trained on extremely large datasets can produce transcripts with numbers written in Arabic numerals. This creates a need for fair comparison with models that output verbatim texts… 9 arXiv — NLP / Computation & Language research 24d ago CASCADE Against Jailbreaks: Combination Across Stages with Controlled Attack-Defense Evaluation arXiv:2609.21793v1 Announce Type: cross Abstract: Defenses against jailbreak attacks on Large Language Models (LLMs) operate at different pipeline stages, such as input modification or output guard, but it remains unclear which defenses to deploy at each stage and how to combine… 35 arXiv — NLP / Computation & Language research 24d ago VQ-Logits: Compressing the Output Bottleneck of Large Language Models via Vector Quantized Logits arXiv:2505.10202v2 Announce Type: replace Abstract: Large Language Models (LLMs) have achieved remarkable success but face significant computational and memory challenges, particularly due to their extensive output vocabularies. The final linear projection layer, mapping hidden… 9 arXiv — NLP / Computation & Language research 24d ago Semantic Calibration Prevails Where Token Confidence Fails: Benchmarking Long-Form Scientific QA arXiv:2602.00279v2 Announce Type: replace Abstract: Reliable uncertainty quantification (UQ) is essential for safe deployment of large language models (LLMs) in scientific question answering, where long-form outputs exceed practical human verification at scale. We introduce the… 17 arXiv — NLP / Computation & Language research 24d ago JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems arXiv:2604.23478v3 Announce Type: replace Abstract: Large language models are widely used to judge the output of other language models, yet whether a judge returns the same verdict when the same request is worded differently remains largely unexamined. We study that question… 23 llama.cpp releases dev-tools 24d ago b11065 CUDA: tune FA for Gemma 4 on Ampere or newer ( #29152 ) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48802880 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS… 28 r/LocalLLaMA community 24d ago The bear can dance: Qwen 3.8 27B on one 3090 for 3 weeks TL;DR: Local agent loop, ~21 days, one RTX 3090. Task was pretty much "build a CUDA inference engine for optimized for yourself on this GPU arch." Got working kernels and benches, not a win over llama.cpp. ~12 human messages. Compaction ate ~83 hours. Old joke: you don’t… 16 Hacker News — AI on Front Page community 24d ago Samsung is expected to more than double output of its HBM4 and HBM4E DRAM Article URL: https://en.sedaily.com/finance/2026/09/20/samsung-to-double-hbm4-output-next-year-sources-say Comments URL: https://news.ycombinator.com/item?id=49778029 Points: 283 # Comments: 190 25 The Information — AI news-outlet 24d ago How Nvidia Is Trying to Solve the Data Center Power Bottleneck Nvidia is tracking “every single gigawatt of land, power and shell around the world, literally everything on the planet,” CEO Jensen Huang said at Goldman Sachs’ annual tech conference earlier this month. “We know where everything is.” Why track power so intently? Nvidia sees… 33 r/LocalLLaMA community 24d ago focus-llama: a llama.cpp fork implementing Declarative Attention (arXiv:2609.02737) I forked llama.cpp to implement Declarative Attention (arXiv:2609.02737, Google DeepMind and KAIST AI). The model declares in its own output which context chunks it needs <focus magic\_chunks="N">), and the engine listens and restricts what the following tokens can attend to. No… 9 llama.cpp releases dev-tools 24d ago b11062 CUDA: enable sparse fa for qwen4 ( #28770 ) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48739244 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:… 35 r/LocalLLaMA community 25d ago Finally got Qwen 3.8 Next running on my v100 6gpu setup (TP2 PP3) Hey everyone, After wrestling with hardware and engine issues for days, I finally got Qwen 3.8 Next running properly on my multi-GPU rig. Thought I’d share the setup journey, benchmarks, and thermal results for anyone trying something similar. Seeing all the ongoing memes on… 16 r/LocalLLaMA community 25d ago Radeon RX 10800 XT can outperform the RTX 5090 by 15-25% in 4K gaming and local AI https://en.gamegpu.com/news/zhelezo/radeon-rx-10800-xt-mozhet-obojti-rtx-5090-na-15-25-v-igrakh-v-v-4k-i-lokalnom-ii The more competition, the better!   submitted by   /u/Lumpy_Phase_9539 [link]   [comments] 11 r/LocalLLaMA community 25d ago I turned an asymetric pair of Tesla V100s PCIe both (16 GB + 32 GB) into a surprisingly capable local LLM lab — 1.38k prompt tok/s, 40 decode tok/s with qwen3.8 27B Q6 and Q8... TL;DR: I run a mismatched Tesla V100-PCIE pair—one 16 GB card and one 32 GB card, 48 GB total—in a Proxmox/LXC-based local-inference lab. The practical winner so far is a recent CUDA build of llama.cpp with tensor split, Flash Attention, --numa distribute , and large batches. On… 29 r/LocalLLaMA community 25d ago General warning about Clore.AI Hello, I know some of us may be tempted to rent out our expensive GPUs to recoup some of the cost of self-hosting, and it should be obvious that this can be a risky decision. I decided to try hosting my rig on clore.ai briefly to see what kind of revenue it could bring in,… 26 r/LocalLLaMA community 25d ago Built a home server from an old PC with GPU upgrade. Qwen3.8 27B runs at ~30 tokens per second. I needed a relatively simple but acceptable level of AI for working on one project. I didn't have any heavy requests, I just needed to give the AI access to the project files so it could search through them for bugs and stuff. I already had an old computer that I decided not to… 20 Page 9 of 10 · 500 articles ← Newer Older →
