News / #gpu Tag Gpu 500 articles archived under #gpu · RSS Sign in to follow llama.cpp releases dev-tools 5d ago b10612 tests : disable DOTS3NOTE arch test for WebGPU ( #27654 ) Co-authored-by: Stanisław Szymczyk sszymczy@gmail.com Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/42591112 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64,… 26 llama.cpp releases dev-tools 5d ago b10610 ggml : shorten virtual device naming in CUDA and Metal ( #27608 ) ggml : shorten virtual device naming in CUDA and Metal Assisted-by: llama.cpp:DeepSeek-V4-Flash-0731 ggml-metal : build device description at init Assisted-by: llama.cpp:DeepSeek-V4-Flash-0731 cont : naming… 22 Import AI (Jack Clark) community 5d ago Import AI 470: No rights for machines; automating environment generation with SPADE; and building better GPU kernels with Hawkeye Differential acceleration of cyber, math, and AI 5 r/LocalLLaMA community 5d ago GPU Poor - Don't overlook Laguna XS 2.1 As per title. I'm not affiliated with the team behind this model in any way, shape or form. As a GPU poor myself (8 GB VRAM laptop + 12 GB VRAM desktop), I found Laguna to be very promising on my laptop. It runs at 30t/s (60k context) and it one-shot my two usual test prompts :… 19 llama.cpp releases dev-tools 6d ago b10605 mamba2 : Flatten in/out projections to dispatch GEMM instead of GEMV ( #27513 ) mamba2 : flatten mamba2 in/out projections to dispatch gemm instead of gemv mamba2 : remove redundant output reshape Website: https://llama.app Attestations:… 9 The Information — AI news-outlet 6d ago Nvidia to Raise Flagship AI Chip Prices 17%, Server Makers Say Prices for Nvidia’s Grace Blackwell and Vera Rubin chip systems are set to rise about 17%, The Information reported Saturday . The price jump adds to a growing array of cost surprises for cloud providers and other data center developers, including unexpected power delays ,… 21 The Information — AI news-outlet 6d ago Nvidia Invests in Data Center Power Firm, Preps Multibillion-Dollar Perplexity Deal Nvidia acquired a minority stake in Cloverleaf Infrastructure, its third equity investment in rapid succession in firms that secure real estate and electricity access for data centers in the U.S. The move is part of Nvidia’s race to lock up data center capacity for its full AI… 17 Smol AI News news-outlet 6d ago not much happened today **Agent harnesses** are becoming a key optimization focus, with NVIDIA research showing traditional skill checks poorly predict agent usefulness and proposing a new metric called **"Skill Lift"**. Open-source implementations of **persistent and self-modifying agents** like… 27 Smol AI News news-outlet 6d ago not much happened today **OpenAI** announced benchmark results for its custom inference chip **Jalapeño**, showing **1.5–1.9×** better efficiency and **1.7–3.6×** lower latency compared to NVIDIA **GB200/GB300**. Deployment starts by year-end with **Gen 2** and **Gen 3** in development. The chip runs… 9 arXiv — Machine Learning research 6d ago Learning Exact NVIDIA SASS Encoders with $\mathbb{F}_2$ Linear Algebra arXiv:2608.20532v1 Announce Type: new Abstract: NVIDIA provides a SASS disassembler but no public SASS assembler for recent data-center GPUs, limiting controlled machine-code rewriting. We present F2Asm, which learns exact 128-bit SASS encoders from paired disassembly and… 25 arXiv — Machine Learning research 6d ago Faults That Fortify: CNN Adversarial Robustness via GPU Undervolting arXiv:2608.20572v1 Announce Type: new Abstract: Convolutional Neural Networks (CNNs) face a dual challenge: vulnerability to adversarial attacks and prohibitive training cost. Adversarial training is effective but expensive, a burden that grows as learning shifts to the… 15 arXiv — Machine Learning research 6d ago Hidden Axis of Uncertainty: Latent-Posterior Alignment in Graph Neural Networks with Bayesian Output Layers arXiv:2608.20758v1 Announce Type: new Abstract: Bayesian Neural Networks (BNNs) with Bayesian output layers provide a principled and tractable framework for quantifying predictive uncertainty, yet the mechanisms shaping that uncertainty remain unclear. While conventional theory… 31 arXiv — Machine Learning research 6d ago Designing a Robust LLM-Based Evaluation System for Agentic AI in Drug Discovery Through Human Alignment arXiv:2608.21057v1 Announce Type: new Abstract: Agentic large language model (LLM) systems are reshaping scientific workflows in chemistry and drug discovery, but evaluating their open-ended, tool-augmented outputs remains a fundamental bottleneck. Reference-based metrics such… 12 arXiv — Machine Learning research 6d ago COEC: Calibrated Orthogonal-Equivalence Compensation for Structured Pruning of Large Language Models arXiv:2608.21142v1 Announce Type: new Abstract: Structured pruning reduces the size and inference cost of large language models (LLMs) by removing weight columns, but the resulting output error can degrade accuracy. Existing training-free compensation methods use an additive… 16 arXiv — NLP / Computation & Language research 6d ago AsmEvo: Agentic Assembly-Level Optimization of AMD GPU Kernels with Functional Equivalence Verification arXiv:2608.20711v1 Announce Type: new Abstract: High-performance ML systems increasingly rely on GPU kernels whose editable source is unavailable, generated, or too distant from final machine code to expose remaining optimizations. Existing LLM kernel optimizers and autotuners… 31 arXiv — NLP / Computation & Language research 6d ago MentorPulse: Refreshing Cross-Model Latent Guidance for Long-Form Generation arXiv:2608.20927v1 Announce Type: new Abstract: Cross-model latent guidance lets a frozen large mentor encode an input once and a frozen small student generate from the resulting signal. Existing methods keep this signal fixed, assuming it stays useful as the output grows; we… 35 arXiv — NLP / Computation & Language research 6d ago PromptResponse: Optimizing Prompts for LLM Coding Tasks arXiv:2608.21074v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used in research workflows and software development pipelines, yet their output remains sensitive to input prompt variations. This paper presents… 35 arXiv — NLP / Computation & Language research 6d ago Trustworthy RAG: An Evaluation Agent for Detecting Misinformation and Knowledge Poisoning in Generative AI Systems arXiv:2608.21095v1 Announce Type: cross Abstract: Retrieval-Augmented Generation (RAG) grounds Large Language Model (LLM) outputs in external knowledge, but RAG systems usually trust whatever they retrieve, creating a Security-Reliability Gap: high semantic relevance does not… 12 arXiv — NLP / Computation & Language research 6d ago Explaining Intrinsic Moral Self-Correction with Mechanistic Interpretability arXiv:2505.11924v4 Announce Type: replace Abstract: Intrinsic moral self-correction refers to the phenomenon where a language model refines its ethical judgments or aligns its outputs purely through prompting. While effective across diverse tasks, its mechanism remains unclear.… 36 arXiv — NLP / Computation & Language research 6d ago MedRAGChecker: Claim-Level Verification for Biomedical Retrieval-Augmented Generation arXiv:2601.06519v2 Announce Type: replace Abstract: Biomedical retrieval-augmented generation (RAG) can ground LLM answers in medical literature, yet long-form outputs often contain isolated unsupported or contradictory claims with safety implications. We introduce… 15 The Information — AI news-outlet 6d ago Nvidia Discusses Perplexity Investment at $30 Billion-Plus Valuation Nvidia is discussing investing in Perplexity as part of an equity round that would value the AI startup at more than $30 billion, according to people with knowledge of the discussion. The round would be worth billions of dollars and boost the startup’s valuation more than 50%… 14 The Information — AI news-outlet 6d ago Nvidia, Salesforce Are in Spotlight This Week Some of the hardest-working people in tech right now have to be the corporate communications folks at Nvidia, which is pretty much never out of the news. Aside from updates on its latest AI chips, Nvidia seems to be investing in almost every part of the AI sector, from data… 26 r/LocalLLaMA community 6d ago Nvidia Customers Notified About AI-Related Price Hikes Above 15%   submitted by   /u/fallingdowndizzyvr [link]   [comments] 13 r/LocalLLaMA community 6d ago Ling Tiny, King of Speed Ling Tiny has now replaced Gemma4-12B in my rig as an auxiliary model doing hindsight operations. This is on a 4060Ti, which is a reasonable GPU available out there, and the speed is phenomenal. Don’t enable MTP, set up the vLLM fork for BailingMoE3. Hope this is useful to… 27 llama.cpp releases dev-tools 6d ago b10594 common : skip device_info loop if it's not going to be printed ( #26692 ) The device_info loop iterates over the discovered devices and gets the available and total memory counts. With the CUDA backend (and possibly others too) this requires creating a GPU context, which, in… 11 r/LocalLLaMA community 6d ago I trained a game music generator I trained a instrumental game music generator. The 1.2B DiT was trained on 1 cloud H100 from scratch in 8 days; I used the VAE from Stable Audio 3. https://huggingface.co/Localsong/Localsong https://huggingface.co/Localsong/Localsong/tree/main/samples… 23 r/MachineLearning community 6d ago 28 TPS on Qwen2.5-7B across two separate cloud regions over public WAN using speculative decoding + CUDA Graphs [P] been building ShardFlow for the past few months, a distributed LLM inference framework that splits any HuggingFace transformer across N GPU machines and uses neural speculative decoding to deal with WAN latency. the setup for the benchmark: two T4 nodes in separate GCP regions… 7 r/LocalLLaMA community 7d ago I hosted Kimi K3 (2.8T parameters) using 8 B300s. 92 tok/s, $190 per million tokens What I ran: 8x B300 on Modal, $56.79 per hour, vLLM, tensor parallel 8, native MXFP4 Cold boot ~27 min (1.56 TB load, JIT, 51 CUDA graph captures) TTFT 0.92 to 1.02 s, decode 92 tok/s steady, 83 tok/s average over 4 prompts $190 per million output tokens. One clean run is about… 10 llama.cpp releases dev-tools 7d ago b10589 cuda : add POOL_1D support ( #27573 ) cuda : add POOL_1D support fix: add missing trailing newline for editorconfig compliance Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/42401257 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple… 29 r/LocalLLaMA community 7d ago Nvidia Poolside deal to compete with Chinese Open Weights Nvidia is investing $1 billion in Poolside and paying $6 billion to license its technology and hire most of its engineers. Over 100 Poolside staff will move to Nvidia to work on Nemotron. Good news for us!   submitted by   /u/mrgreatheart [link]   [comments] 38 r/LocalLLaMA community 7d ago Has anyone actually made 64k feel like 300k+ with recursive local agents? I'm running Qwen 3.8 27B locally on a single GPU. I can push the context to 131k, but I'd rather run it faster at 64k if the agent can manage context properly. What I have in mind is pretty simple: one model stays loaded the whole time main agent gets 64k when something is too… 32 The Information — AI news-outlet 7d ago Nvidia AI Chip Prices to Rise About 17%, Server Makers Tell Customers Prices for some of Nvidia’s flagship AI server chip systems are set to rise about 17%, according to two people whose firms were notified by server makers that provide such systems. The price hikes generally apply to Grace Blackwell 300 and Vera Rubin 200 chip systems that are… 14 r/LocalLLaMA community 7d ago Watching that wattage, in your terminal. Released today: version 1.3 of energygraph Zero build dependencies, lightweight tool for live views of the power-consumption. Version 1.3 adds support for dGPUs from nvidia, intel, amd. Depending on vendor support, you can also get the consumption by your cpu cores, iGPU,… 4 r/LocalLLaMA community 8d ago Need support for llama.cpp with multi GPU Using llama.cpp I seem to be unable to get my to GPUs working tougether correclty, so I need help somehow. Setup: 96GB RAM, one Blackwell 5000 (48GB) and one 3090 (24GB). I am trying to run the UD-Q3_K_XL quant of Deepseek4 flash which has about 120GB size. Using just the… 20 r/LocalLLaMA community 8d ago Sharp template to NInfer: -42% output tokens, same speed Sharp is u/peculiar-ragdoll 's system prompt that makes Qwen answer way more tersely without losing correctness. It's built on top of froggeric's fixed chat templates for Qwen; several fixes now in the v22.x templates (error-escalation tiers, false retry-loop kills, multi-system… 20 r/MachineLearning community 8d ago I developed my own quantized LLM from scratch, trained on 30B tokens, deploys in 60 MB [R] I trained a 250M parameter model from scratch on 30B tokens of fineweb. It’s quantized to under 2 bits so the whole deployment is 60 MB and it needs about 80 MB of RAM to run. Runs around 400 tok/s on a normal laptop CPU, no GPU needed. How the long context works: the most… 10 r/LocalLLaMA community 8d ago Qwen 3.8 27b - PI AGENT vs OPENCODE - another smaple That is the second comparison and the last one. I will not be spamming again ;) Continuation from: https://www.reddit.com/r/LocalLLaMA/comments/1vu0u2v/qwen_38_27b_pi_agent_vs_opencode/ That is one of my many tests I make comparing output quality. What is more interesting using… 9 The Information — AI news-outlet 8d ago Exclusive: OpenAI Explored Stake in Stargate Power Developer Lancium OpenAI explored its own investment in or acquisition of Lancium, the power developer behind its Stargate facility in Abilene, Tex., last quarter, The Information reported Friday . Ultimately, Nvidia invested billions of dollars in the firm . The OpenAI interest shows the length… 22 TechCrunch — AI news-outlet 8d ago Nvidia partners with data center developer Cloverleaf Nvidia continues to pour money into data center development — just as AI data centers bring lots of money into Nvidia. 9 The Information — AI news-outlet 8d ago Nvidia is Using Land and Electricity Deals to Lock In Its Hardware Bundle Nvidia is racing to lock up data center capacity for its AI hardware before its rivals do. On Friday, the company announced it had acquired a minority stake in Cloverleaf Infrastructure, its third equity investment in rapid succession in firms that secure real estate and… 25 TechCrunch — AI news-outlet 8d ago Nvidia just showed that the harness, not the AI model, is now the real hero Nvidia research shows that AI agents can perform well, and not go off the deep end, through fine-tuning, even if the AI model isn't that great at the task. 28 r/LocalLLaMA community 8d ago Qwen3.8-27B on an RTX 5060 Ti 16GB: IQ4 vs Q8, 64K context, MTP, vision, and agent benchmarks I’ve been testing Qwen3.8-27B as a possible replacement for the Qwen3.5-9B that I have been running on RTX 5060Ti 16G. The goal was not just maximum tokens/sec, but useful context capacity, reliable tool calling, multi-turn behavior, and vision on a single 16GB GPU for true… 34 r/MachineLearning community 8d ago Does telling an LLM to "be concise" actually save you money? We measured it across 9 models. Compressing the output can save you money and keep accuracy, compressing the input prompt does not. [R] LLMs are too verbose and with a black box model the only things you control are what goes in and how you tell it to write back. Yesterday Claude Code shipped a "concise output style" where Claude keeps things short. We already have a paper out about this! We tested both… 7 r/MachineLearning community 8d ago I have a mid-sized GPU cluster and was thinking about giving free compute [D] I have built an on-prem GPU cluster, 8 nvidia 16GB GPU's and 256GB CPU RAM, 50TB HDD and several TBs of SSDs. I have used it, and currently use it, for ML/AI research. But that research is not constantly running jobs, sometimes I use it heavily and other times it's idle. I was… 12 NVIDIA Developer Blog official-blog 8d ago GPU-Accelerated Clustering for Financial Instruments at Scale Use AdaptGrow, a GPU-accelerated matrix factorization algorithm, to turn rolling correlation and tail-dependence matrices into hard clusters, soft factor... 23 Simon Willison community 8d ago Stop Making TUIs Stop Making TUIs Thomas Ptacek advocates for building real native user interfaces for even the smallest of personal tools, because coding agents have reduced the cost of getting a usable-enough GUI up and running to almost nothing. I wrote about my vibe-coded bandwidth and GPU… 27 r/LocalLLaMA community 8d ago Ultrafast Qwen3-TTS at 34 ms Time-to-First-Audio, Handling 10 Requests Per Second [OSS] Hey locallama! We recently open sourced a Qwen3-TTS 1.7B implementation that achieves 10 requests per second (RPS) and sub-50 ms p95 time-to-first-audio (TTFA) while maintaining real-time playback on 1 x H100. This extends to 20 RPS at sub-100 ms p95 TTFA. By adjusting settings,… 31 NVIDIA Developer Blog official-blog 8d ago Maximizing AI Factory Performance per Watt with NVIDIA DSX MaxLPS AI factories are power-constrained industrial systems. The question is no longer how many GPUs fit in a data center, but how much AI output each available... 9 r/LocalLLaMA community 8d ago NVIDIA AVO got 100% on ARC-AGI-3. It completed all 183 levels across all 25 public environments, figuring out what to do with no instructions, explicit rules, or stated goals.   submitted by   /u/theologi [link]   [comments] 35 NVIDIA Developer Blog official-blog 8d ago NVIDIA AVO Reaches 100% on ARC-AGI-3, Demonstrating a Frontier-Level General-Purpose Architecture for Long-Horizon Autonomous Agents A frontier language model is only one component of an AI agent. The surrounding agent system—often called a harness—determines how the model receives... 18 Page 3 of 10 · 500 articles ← Newer Older →
