News / #gpu Tag Gpu 500 articles archived under #gpu · RSS Sign in to follow llama.cpp releases dev-tools 10d ago b11391: cuda : move blocks_per_col to where it is used (#29939) Signed-off-by: Adrien Gallouët angt@huggingface.co 17 llama.cpp releases dev-tools 10d ago b11390 CUDA: fix MMQ memory fault if n_expert >> n_ubatch ( #29941 ) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52581427 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS… 9 llama.cpp releases dev-tools 10d ago b11387: spec : fix n-gram drafts rejected at temp > 0 after truncation (#29924) Co-authored-by: Pranesh Gonegandla pgonegandla@nvidia.com 7 llama.cpp releases dev-tools 10d ago b11382 webgpu: add f16 support to fill/set_rows ( #29897 ) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52497832 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS… 15 r/LocalLLaMA community 10d ago Optimal settings for 2 GPUs in LM Studio Hello everybody. I've got a 5070ti and a 5060ti both 16GB in my system which is a 5900X and 64GB DDR4 RAM. I'm trying to run some 16-18GB models like Qwen, Cydonia, Skyfall. I'm having problems utilising the VRAM I have to get the best usage out of it. LM Studio sees the 32GB… 8 r/LocalLLaMA community 11d ago Dual Radeon MI50 benchmarks Still don't have a good cooling solution, but here are few benchmarks. I lowered the power limit (TDP) to 145 watts each. I changed the firmware on one MI50 to activate the miniDP port. Did have to use xrandr to create a new mode so I could get 1920x1080 output. Each GPU has… 25 r/LocalLLaMA community 11d ago Running Qwen3.8 Flash Next 176B on a 16GB RTX 3080 Laptop + 32GB RAM + SSD I wanted to see how far I could push a fairly ordinary laptop with a huge MoE model. Turns out, Qwen3.8 Flash Next 176B can run on: RTX 3080 Laptop — 16GB VRAM 32GB system RAM SSD No 128GB/256GB RAM workstation and no multi-GPU setup. I’m running it with TensorSharp , my… 36 Hacker News — AI on Front Page community 11d ago The work by Valve's Timur Kristóf on improving old AMD GPUs on Linux Article URL: https://www.phoronix.com/news/XDC-2026-Valve-Timur-AMDGPU Comments URL: https://news.ycombinator.com/item?id=49946895 Points: 398 # Comments: 67 26 r/LocalLLaMA community 11d ago I built Ninfer 4080 for 16GB class GPUs Hi everyone, TL/DR I created NInfer 4080 to run ISTA-DASLab-Qwen-3.8-27B-GSQ at 100k context on an RTX 4080 16GB GPU using way more of the hardware capabilities ( max overall: 2720 tok/s prefill, 262 tok/s generation ) and sharing it with the community now so others can also… 6 r/LocalLLaMA community 11d ago Qwen Flash next on 64GB RAM unified iGPU anyone ? (non-mac) I've seen people reporting running it with 12GB VRAM + 64 GB RAM. Also with 64 GB RAM unified in Macs, but I was wondering if it's possible with any inference backend to run it for example on a 64 GB RAM minipc+ iGPU (780m in my case). I'm currently running 125B Ling 3.0 flash… 13 llama.cpp releases dev-tools 11d ago b11374 ggml-openvino: update to 2026.4.1, optimize performance, expand ops, improve device listing. ( #29852 ) ggml-openvino : Qwen3.5 MoE perf ( #312 ) Squash of ravi9#312 : ggml-openvino: add detailed inference profiling (Yu, Zijun) ggml-openvino: use remote output tensors by default… 16 r/LocalLLaMA community 11d ago Local Ai Pc 7663 Dual Epyc / Dual 9709 CPU Information Name AMD EPYC 7663 Topology 2 Processors, 112 Cores, 224 Threads Memory Information RDiMM 2933Mhz Size 1007.61 GB System Information Operating System Ubuntu 24.04.5 LTS Motherboard Giga Computing MZ72-HB2-00 GPUs 2x ASUS Turbo R9700 AI Pro 32 GB newest BIOS low… 11 r/LocalLLaMA community 11d ago Qwen3.8-Flash-Next 177B running at 11–15 tok/s on a single RTX 5070 12GB + 32GB RAM DDR4 Benchmarking an LLM here with a NVIDIA RTX 5070 12 GB VRAM here I had been working on a llama.cpp based expert streaming setup for Qwen3.8-Flash-Next 177B (UD-IQ3_XXS) on Windows. Benchmark is about 11.5 tok/s, up from roughly 7 tok/s on the inherited setup. In normal… 35 llama.cpp releases dev-tools 11d ago b11370 CUDA: fuse shared experts into MMVQ ( #29184 ) CUDA: fuse shared experts into MMVQ check if buffer is null move stride_col_dst to fusion args Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52371643 macOS/iOS: macOS Apple Silicon… 36 arXiv — Machine Learning research 11d ago Fast Polynomial Transcendentals for LLMs arXiv:2610.00049v1 Announce Type: new Abstract: Graphics processing unit (GPU) generations scale matrix, special-function, and memory pipelines at different rates, so kernel bottlenecks move as hardware evolves. FlashAttention-4 exposed this imbalance inside attention on NVIDIA… 27 arXiv — Machine Learning research 11d ago Verification Pulses and the Cost of Escaping Wrong Consensus arXiv:2610.00256v1 Announce Type: new Abstract: External verification can correct individual outputs while leaving a self-reinforcing population in the basin of a wrong consensus. We study how the timing and addressing of a fixed verification budget affect recovery in an… 12 arXiv — Machine Learning research 11d ago Score the Update, Not the Token: Descent-Aligned Routing for Combinatorial LoRA Experts arXiv:2610.00493v1 Announce Type: new Abstract: Mixture-of-LoRA-experts methods raise the capacity of low-rank adaptation by routing each token to a few low-rank experts. Nearly all of them tie one input-side factor to one output-side factor per expert, and nearly all of them… 6 arXiv — Machine Learning research 11d ago Beyond Affine Transformations: A Soft Dominance Layer for Coordinate-Wise Neural Computation arXiv:2610.00563v1 Announce Type: new Abstract: This paper presents a preliminary study of an alternative to the affine transformation underlying conventional neural-network layers. In the proposed Soft Dominance Layer, each output unit compares input coordinates with a… 35 r/LocalLLaMA community 11d ago Anyone sitting on a lot of slow system memory and a modest GPU.. try Strata + Qwen3.8 Next. IQ3_XXS weights are just under 80GB and my slowww DDR4+7900XTX is stabilizing around 45-70/s (sometimes higher while coding depending on mtp). Looking online I'm seeing similar results for users with 12GB and 16GB cards, and significantly faster numbers for owners of DDR5. (In… 22 llama.cpp releases dev-tools 11d ago b11365 ggml-cpu : fix soft_max_back wrong output when dst aliases src1 ( #27096 ) ggml-cpu : fix soft_max_back wrong output when dst aliases src1 GGML_OP_SOFT_MAX_BACK is listed in ggml_op_can_inplace, so the graph allocator may assign dst to alias either src0 (dy) or src1 (y). The… 10 llama.cpp releases dev-tools 12d ago b11355 vulkan: disable large matmul tile on Samsung GPUs with 32KB shared memory ( #28531 ) Assisted-by: Claude Opus Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52313922 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64,… 13 r/LocalLLaMA community 12d ago microsoft/FrogNano-4B-2609 · Hugging Face An agentic model from Microsoft for the GPU poor https://huggingface.co/bartowski/FrogNano-4B-2609-GGUF FrogNano is derived from Qwen/Qwen3.5-4B, a general-purpose post-trained model designed for language, reasoning, coding, agentic, and multimodal tasks. FrogNano inherits… 22 Ars Technica — AI news-outlet 12d ago US arrests tech CEO accused of smuggling $300M in Nvidia chips into China Nvidia’s chip-smuggling problem won’t go away as arrests continue. 14 The Information — AI news-outlet 12d ago Lambda Secures $1 Billion GPU Loan AI cloud company Lambda said it has secured its first delayed draw term loan at over $1 billion to purchase more than 30,000 Nvidia graphics processing units. The fixed-rate facility carries an interest rate of 6.78%. Privately held Lambda said in an announcement late on… 22 r/LocalLLaMA community 12d ago I made my iPhone a second GPU for my 24 GB MacBook: Qwen 3.8 27B prefills 29–44% faster & my holds part of the CTX window. **DISCLAIMER** THE PREFILLING TPS SHOWN ON THE PHONE IS COMPUTED ONLY FOR THE LAYERS IT HOLDS. ALREADY FIXING IT TO SHOW END-TO-END PREFILL RATE. NUMBERS BELOW ARE ACCURATE FOR E2E PREFILL RATE. Every file or tool result my agent reads on a 24 GB M4 Pro MacBook is a wait, and… 33 Ars Technica — AI news-outlet 12d ago The 7-year-old Nvidia Shield TV is now $100 more expensive thanks to AI Initially launched in 2019, the Nvidia Shield TV Pro now retails for $299.99. 18 r/LocalLLaMA community 12d ago Tuned/abliterated Qwen3.8-27b into a 24gb card 262k guff using the newest unreleased version of LexiPanel. It's fast with reliable draft acceptance. Made for 7900xtx but should work on whatever 24gb card with this setup and headless. Doesn't get dumber while coding like most of… https://huggingface.co/Wa1k3r/Qwen3.8-27b-CODER-4q_xs-24GB-262k-Optimalcardfit Qwen3.8-27B CODER — IQ4_XS imatrix · 24 GB card fit · ~262k context · MTP draft Quantized, Abliterated, and fitted by LexiPanel. Its Fit planner chose the tensor mix to fill one 24 GB GPU at about… 30 r/LocalLLaMA community 12d ago Two 96 GB Ascend cards crun Qwen3.8-flash-next hardware notes, vLLM work, benchmarks, and what is next I have been building a somewhat unusual local inference machine around two Huawei Atlas 300I Duo cards. They are relatively inexpensive, passive, dual-accelerator PCIe cards with 96 GB of device memory apiece. They are also absolutely not drop-in CUDA replacements. When I first… 36 llama.cpp releases dev-tools 12d ago b11344 CUDA: fix 2 broken Volta FA cases ( #29803 ) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52093363 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework… 9 The Information — AI news-outlet 12d ago California Man Arrested for Smuggling Nvidia Chips to China U.S. authorities arrested a California man on Thursday for smuggling more than $300 million worth of servers containing restricted Nvidia AI chips to China. Greg Lui, the owner of Earthmade Computer Inc, a privately held company based in San Gabriel Valley, allegedly worked with… 32 The Information — AI news-outlet 12d ago Chinese State-Backed Firm Funded Purchase of Nvidia Chips A state-backed Chinese financing company has revealed in documents filed with regulators that it funded the purchase of Nvidia chips subject to U.S. export restrictions, Bloomberg reported on Friday. Semi-Tech Leasing Group, owned by several local government entities, has… 18 arXiv — Machine Learning research 12d ago Alignment via Training Against Probes Without Losing Monitorability arXiv:2609.38645v2 Announce Type: new Abstract: Models are usually aligned based on their observed outputs, using demonstrations, preference data, or reward signals. These objectives reward responses that look aligned. More capable models may learn to satisfy them without… 23 arXiv — Machine Learning research 12d ago dattri-LLM: A Unified and Efficient Library for Training Data Attribution at LLM Scale arXiv:2609.38767v2 Announce Type: new Abstract: Training data attribution (TDA) estimates the contribution of individual training examples to model outputs. Most scalable TDA methods rely on per-example gradients, whose computation and use at LLM scale pose challenges in… 29 arXiv — Machine Learning research 12d ago SparLeak: Privacy Leakage from Sparse Attention in LLM Inference on Shared GPUs arXiv:2609.38830v1 Announce Type: new Abstract: Sparse attention is widely used to accelerate long-context inference in modern large language models (LLMs), but its input-dependent execution behavior introduces previously unexplored privacy risks. We identify a new GPU… 33 llama.cpp releases dev-tools 12d ago b11335 cuda : route sm70 to the Turing MMVQ nwarps table ( #29753 ) cuda : route sm70 to the Turing MMVQ nwarps table Volta (sm_70) has no MMVQ parameter table of its own and falls through to GENERIC, which launches K-quant batch-1 decode (ncols_dst == 1) at nwarps=4. sm_70 shares… 5 ThursdAI news-outlet 12d ago ThursdAI - Oct 1 - OpenAI joins the assistant race, CoreWeave drops serverless GPUs & more Listen now (117 mins) | From CoreWeave: GPT-6.1 Sol is near-Astra for a fifth of the price, Sonnet 5.5 beats Opus on Terminal-Bench, Gemini 4 Argon tops the charts, and we went LIVE from Fully Connected. 38 llama.cpp releases dev-tools 12d ago b11333 webgpu: add bfloat16 support for MUL_MAT/MUL_MAT_ID/GET_ROWS- #29358 ( #29358 ) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52019658 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED… 15 llama.cpp releases dev-tools 12d ago b11331 CUDA: Handle compute type for NVFP4 on cublass path ( #29173 ) CUDA: Handle compute type for NVFP4 on cublass path Signed-off-by: ynankani ynankani@nvidia.com Use BF16 compute type for quantized models if HW allows Signed-off-by: ynankani ynankani@nvidia.com Set acc prec to bf16… 26 llama.cpp releases dev-tools 13d ago b11324 llama-mmap : avoid a second full-size copy of each tensor with direct-io ( #29749 ) Assisted-by: Claude Co-authored-by: Pranesh Gonegandla pgonegandla@nvidia.com Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51929570 macOS/iOS: macOS… 5 r/LocalLLaMA community 13d ago Now which lab is behind fledge alpha? New free/trial model dropped on OC. Details are yet to be provided. Could this be deepseek v4.1 pro?   submitted by   /u/Gohab2001 [link]   [comments] 16 The Information — AI news-outlet 13d ago Exclusive: Nvidia, SoftBank Make Final $20 Billion Investment in OpenAI’s Last Round Nvidia and SoftBank have each made the final $10 billion investment in each of their $30 billion pledges to OpenAI’s last funding round, according to a person with knowledge of the investments and a statement from SoftBank. OpenAI in March said it had $122 billion in commitments… 20 NVIDIA Developer Blog official-blog 13d ago Build Applications on NVIDIA BlueField Faster with NVIDIA DOCA Agent Skills AI agents are becoming a standard part of development workflows, but general-purpose agents weren't built with specialized infrastructure software such as... 24 NVIDIA Developer Blog official-blog 13d ago Build Local AI Apps with C++ and NVIDIA TensorRT RTX Samples Adding AI models to local applications requires a portable model format, a reliable runtime, and acceleration that works across target systems. Do Inference Now... 22 r/LocalLLaMA community 13d ago What's the best setup for Qwen3.8 27b for a 16 gig VRAM? Hello guys! I have been experimenting with qwen 3.8 for a long time and I hadn't been able to get reasonable speed. I am on a 5060Ti 16 GB, and although this gpu can game, I am aware that AI demands more than 16 GB. I am on a Fedora 44, AMD Ryzen 9600x and 16 GB system ram (16… 36 The Information — AI news-outlet 13d ago Sharon AI Announces $356 Million GPU-Backed Loan Australian neocloud Sharon AI has secured $356 million through its first loan backed by its graphics processing units, making it the latest cloud provider to finance their expansion by borrowing against chips and cash flows from customer contracts. The senior secured facility… 25 r/LocalLLaMA community 13d ago $5400 eBay 8x V100 server cranks on flash-next Had opus 5.5 do the bring up.. super solid results. Using only 4 GPU results in KV of like 120k with image on. 27B TP=4 results in >200 tok/s with dflash, prefill around 2.5-3.5k. Both these are running the nvfp4 Nvidia checkpoints. Found a magic repo that unpacks into fp16 on… 22 llama.cpp releases dev-tools 13d ago b11320 common : add LLM-jp-4.1 Harmony dialect handler ( #29681 ) LLM-jp-4.1 uses the GPT-OSS format, but its tokenizer decodes a space after every special token and parallel tool calls are separated by <|end|>. The GPT-OSS handler rejects this output, so add a dedicated handler,… 21 r/LocalLLaMA community 13d ago Dual GPUs vLLM resources for 50 + 40 nvidia GPUs i wanted to see how far i can push my PC and on the way i discoverd the resources are limited so i made my own and i hope it helps . https://github.com/ExTV/dual-gpus-vllm   submitted by   /u/Fz1zz [link]   [comments] 30 r/MachineLearning community 13d ago Gemini 4 Argon - 1 Million Output Headroom. Hype or a Leap? [D] I rarely write about benchmarks; a competitor always beats them next week. But I care about 'Leaps'. Gemini 4 Argon feels like one to me. While Opus 5.5 and Astra cap output at 128-300K tokens (~90-180 pages), Argon hits 1 Million (~1400 pages). "Context glue" ruins agentic… 4 Latent.Space news-outlet 13d ago [AINews] Gemini 4 Argon: GDM’s answer to Astra/Fable, with 1M output ... but you can’t try it yet unless you are “government users and trusted cyber defenders in the Fairwind Program” 33 Page 5 of 10 · 500 articles ← Newer Older →
