News / #gpu Tag Gpu 500 articles archived under #gpu · RSS Sign in to follow r/LocalLLaMA community 13h ago Any current Voice2Voice AI model that runs locally that’s good? You guys remember sesame AI? With their really good AI voice model? Obviously ChatGPT has their voice model that’s also really good. Is there any smaller local variant that runs on like consumer grade gpu‘s (12,16 24gb?) I think NVidia released something but I didn’t really… 25 llama.cpp releases dev-tools 14h ago b10687 opencl: use a better matmul path on two Adreno GPU generations ( #27640 ) opencl: default the Adreno xmem F16xF32 GEMM on for X2E kernel_mul_mm_f16_f32_l4_lm is the slowest matmul this backend has on Adreno: on the X2-90 it runs the gpt-oss-20b attention projections at roughly a… 27 r/LocalLLaMA community 14h ago (NInfer Fork) I wanted to have a 1M context Qwen-3.8 27B, tp2, dual 5090s Hey! I forked NInfer (a from-scratch C++20/CUDA inference engine for Qwen models) and added two things: tensor-parallel across two GPUs, and YaRN ×4 rope scaling. Together they let Qwen3.8-27B NVFP4 run a 1,048,576-token context on two consumer 5090s — 27.4 GB per card, no… 21 TechCrunch — AI news-outlet 19h ago Nvidia’s AI advantage is moving beyond the GPU The new generation of data center systems is increasing efficiency with smarter traffic control instead of just more processor cycles. 9 r/LocalLLaMA community 19h ago Qwen 3.8 27B at 50 tok/s with 100k Context on a 16GB GPU! (beellama.cpp) I wanted to share my successful setup for running a Qwen 3.8 27B model with a massive context window on a consumer 16GB GPU (RTX 4070 Ti SUPER). The goal was to fit everything into VRAM without sacrificing quality or speed. 🧠 Key Components Model:… 38 r/LocalLLaMA community 1d ago Qwen3.8-Flash-Next + MTP on Strix Halo: Vulkan Runtime Notes Below are the benchmark results for running Qwen3.8-Flash-Next on Strix Halo using the Vulkan backend of llama.cpp, combined with MTP model. Hardware Item Details CPU AMD Ryzen AI MAX+ 395 (16C/32T) GPU Radeon 8060S (integrated, RADV STRIX_HALO) RAM 128GB unified memory Software… 27 r/LocalLLaMA community 1d ago 50% tg increase with offloading "hot" experts to VRAM I got a 50% performance boost (20 t/s -> 30 t/s) in llama.cpp for MoE models that don’t fit entirely in VRAM—in my case, Qwen 3.8 Flash Next. The idea is simple: instead of offloading entire layers to the GPU, I offload only the “hot” experts. I found that certain groups of… 34 TechCrunch — AI news-outlet 1d ago Neocloud Lambda secures $1B in debt to buy more chips Neocloud Lambda has raised $1B in private debt to buy Nvidia AI chips and lease them to Microsoft. It's the latest in a string of loans, underscoring the high cost of the AI boom. 17 r/LocalLLaMA community 1d ago Breeze-TTS-2 initial impressions: genuinely 'frontier' TTS You can test it out on breezblue's playground or use it locally, its only ~7GB.   submitted by   /u/Gohab2001 [link]   [comments] 20 NVIDIA Developer Blog official-blog 1d ago Deploy an Open Model from Checkpoint to Inference in Two Commands with NVIDIA TensorRT Model Connect Open AI models are evolving faster than ever, but bringing them into native applications can still require model-specific conversion, preprocessing,... 22 r/LocalLLaMA community 1d ago Qwen3.8-Flash on RTX3090 + 64GB RAM (but you only need 12GB VRAM) I've got Qwen3.8-Flash-next running on RTX 3090, Ryzen 9 3950X, a PCIe 3.0 motherboard, and 64GB DDR RAM from 2020. IQ4_XS weights, full kvarn5 context, vision on GPU, experts in host RAM, n-grams on disk. MTP works but actually slows decode down even with 80% draft acceptance,… 18 llama.cpp releases dev-tools 1d ago b10673 metal : add fa-vec tunings for M4 ( #27875 ) This adds fa_vec_tuned_table records for Apple M4 to ggml-metal-tuning.cpp. Includes F16, Q4_0, Q4_1, Q5_0, Q5_1, and Q8_0. (M4, 10 GPU Cores) Co-authored-by: Strongtut 8432058+Strongtut@users.noreply.github.com Website:… 13 r/LocalLLaMA community 1d ago Heat! As in actual physical heat. How are people coping with the heat that running a decent inference rig pumps out? I've got dual 5060 Ti GPUs and a relatively modest CPU (Intel 14 Core Ultra 5 245KF Desktop on an Msi MPG Z890 motherboard) and if I use it for a coding session or… 8 llama.cpp releases dev-tools 1d ago b10668 metal : add fa-vec tunings for M3 Max, M5 and M5 Pro ( #27863 ) metal : add fa-vec tunings for M5 This is a followup contribution to efeda76 as requested in https://github.com/ggml-org/llama.cpp/discussions/27668 to add support for additional Apple GPUs. I generated this output… 22 llama.cpp releases dev-tools 1d ago b10667 metal : add fa-vec tunings for M4 Pro ( #27824 ) This is a followup contribution to efeda76 as requested in https://github.com/ggml-org/llama.cpp/discussions/27668 to add support for additional Apple GPUs. I generated this output using the provided instructions: git clone… 36 arXiv — Machine Learning research 2d ago CG4AI: A Column Generation Framework for Training AI Models Under Constraints arXiv:2608.26375v1 Announce Type: new Abstract: Standard machine-learning training minimizes a loss function over a dataset, but does not guarantee that the resulting model will satisfy predefined rules or constraints on its outputs. In many real-world applications, ranging from… 30 arXiv — Machine Learning research 2d ago Diffusion Policies for Short-Horizon Planning in Robot Crowd Navigation arXiv:2608.27158v1 Announce Type: new Abstract: Robot crowd navigation requires safe and efficient decision-making under dense, dynamic, and multimodal human--robot interactions. Existing reinforcement-learning methods typically output a single reactive action at each timestep,… 10 arXiv — Machine Learning research 2d ago Common Geodesics Do Not Guarantee Fisher Consistency of the Structured SVM: Minimal Counterexamples and a Tree-Metric Classification arXiv:2608.27203v1 Announce Type: new Abstract: A known necessary condition for Fisher consistency of the structured support vector machine requires the task loss to be a metric for which every output triple has a common geodesic point. We show that this condition is not… 29 arXiv — NLP / Computation & Language research 2d ago Self-Generated Text Recognition: Quality Heuristics, Cross-Task Transfer, and Downstream Bias in LLM Evaluation arXiv:2608.26159v1 Announce Type: new Abstract: Self-Generated Text Recognition (SGTR)--the ability of an LLM to identify its own outputs--poses risks to AI safeguards that rely on LLMs as evaluators or monitors. Specifically, an LLM may recognize outputs from other copies of… 34 arXiv — NLP / Computation & Language research 2d ago A Multi-Framework Comparison of Outline Stages in Long-Form Generation with LLMs arXiv:2608.26177v1 Announce Type: new Abstract: Long-form generation exposes fundamental limitations of large language models. Even 70B-parameter models exhibit length collapse at 16k-token outputs, and multi-chapter stories frequently trigger the attribute drift characteristic… 11 arXiv — NLP / Computation & Language research 2d ago Prediction of Prediction (PoP): Inter-Layer Activation Fusion for Single-Pass Hallucination Detection in Large Language Models arXiv:2608.27165v1 Announce Type: new Abstract: Autoregressive large language models (LLMs) routinely generate factually incorrect outputs with high decoding confidence, limiting their deployment in high-stakes workflows. Existing output-stage uncertainty metrics can fail when… 8 r/LocalLLaMA community 2d ago Ornith-1.5-35B-A3B on 8 GB VRAM: I think I've found my sweet spot A few days ago I posted asking what people considered the best local model for an 8 GB VRAM GPU . At the time, my personal sweet spot was Qwen3.6-35B-A3B , for agentic coding with Pi.dev. Well… Thanks to the suggestions in that thread, I think I've found something even better.… 25 r/LocalLLaMA community 2d ago I am Concerned if Nvidia Acquires Llama.CPP, Dev Team and HF, Anybody else? I dont know about others, but Nvidia is aiming (potentially) to close the lid on older GPUs since they want to push their new technology. Llama and team has been the to go places for older GPUs like V100s. Knowing how Nvidia have tried killing these GPUs of relevancy concerns me… 25 ThursdAI news-outlet 2d ago NVIDIA Buys Hugging Face! GLM-5.3-Flash, Qwen4 Preview, Gemini Omni 1.1, and the Datacenter Debate w/ Andy Masley From CoreWeave - join Alex and ThursdAI co-host, covering the last week of the summer in AI, with 4 Flash models, Datacenter debate & more AI news 11 Ars Technica — AI news-outlet 2d ago Report: Nvidia to acquire AI model repository Hugging Face for $13 billion Nvidia is nabbing critical infrastructure for open models as interest grows. 13 r/LocalLLaMA community 2d ago Let’s be real, memory and gpus price will continue go up next year and the year Models will continue improve for open and closed labs, the demand for compute and memory will continue to increase. Expect to pay double or more for ddr 5 and 6 ram and +60% plus for new consumer gpus . Even a 512 gb mac studio will likely be over 22k .   submitted by  … 8 r/LocalLLaMA community 2d ago With HuggingFace, Nvidia is also acquiring llama.cpp and the team behind it With this move Nvidia is not only acquiring the HuggingFace platform, but they might also effectively acquire the copyright to the `llama.cpp` project, together with the entire team behind it. In February 2026 the llama.cpp team was employed by HF in order to continue working on… 37 r/LocalLLaMA community 2d ago NVIDIA Next Gen Vera Rubin GPUs scheduled for mid-2027 Here’s hoping RTX 6090 also comes at MSRP of $6969 lol. With how expensive RTX 5090s and 6000 Pros have gotten it doesn’t sound too far fetched.   submitted by   /u/Leafytreedev [link]   [comments] 13 r/LocalLLaMA community 2d ago friendly reminder you can legally torrent ai models. Repost because reddit keeps thinking this is piracy or illegal. It is neither. A lot of people are skeptical Nvidia will keep huggingface intact now that they will buy huggingface. There's a lot of doom and gloom about not having any alternatives, removing nsfw models, saying… 21 r/LocalLLaMA community 2d ago Nvidia is buying Huggingface for $12.9 billion   submitted by   /u/doomdayx [link]   [comments] 6 r/LocalLLaMA community 3d ago Dflash 2 speculative decoding by SamuelOliveirads · Pull Request #2345 · ikawrakow/ik_llama.cpp Didn't update on ik_llama.cpp updates for sometime. Updates from this month: cuda : repair the HIP build, and validate IQ4_KS and IQ4_KT on RDNA3- #2339 vulkan : add IQ4_KS and IQ4_KT support- #2332 Initial implementation of DSpark model: Ling-3.0 (bailingmoe3) runtime support… 29 The Information — AI news-outlet 3d ago Nvidia Agrees to Buy Hugging Face For $12.9 Billion Nvidia has agreed to buy Hugging Face, the GitHub-like repository of open-source AI models, for $12.9 billion, roughly 80 times the startup’s forward revenue, after deal talks kicked off when another suitor came calling, The Information reported . Salesforce, an investor in the… 37 r/LocalLLaMA community 3d ago NVIDIA buying HF isn't a good thing for open source   submitted by   /u/johnnyApplePRNG [link]   [comments] 18 TechCrunch — AI news-outlet 3d ago Nvidia closes in on Hugging Face acquisition Nvidia has reportedly agreed to buy Hugging Face, the popular open-source AI hub, for $12.9 billion in a move that would let Nvidia both protect its chip empire and jump back into the cloud business. 25 arXiv — NLP / Computation & Language research 3d ago Output Dilution: Redundant but Fragile Representations in MoE Models arXiv:2608.25231v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models appear to encode moral content as robustly as dense models, yet prove far more fragile in their encoding. In OLMoE-1B-7B, linear probes recover moral valence from nearly every expert-layer… 27 arXiv — Machine Learning research 3d ago Prefix-Denoising Consistency: Test-Time Verification for Diffusion Language Models arXiv:2608.25311v1 Announce Type: new Abstract: Diffusion Language Models (DLMs) have recently become increasingly competitive with autoregressive (AR) models, and even outperform them on certain tasks. Unlike AR models, DLMs produce output through iterative denoising without a… 20 arXiv — Machine Learning research 3d ago DualOPSD: Adaptive Privileged Teachers for On-Policy Self-Distillation arXiv:2608.26019v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) uses a privileged copy of the student model to provide dense supervision without an external teacher. OPSD keeps this privileged teacher fixed, even though the student distribution and output… 36 arXiv — NLP / Computation & Language research 3d ago DataKernelBench: Can LLMs Optimize Database Queries on GPUs? arXiv:2608.25061v1 Announce Type: new Abstract: GPUs increasingly accelerate database systems, but query-specific peak performance still often relies on hand-written kernels. Existing LLM kernel benchmarks focus on machine learning operators, leaving irregular, heterogeneous,… 11 arXiv — NLP / Computation & Language research 3d ago TOPAS: Workflow-Aware Prefix-State Scheduling for Multi-Agent LLM Serving arXiv:2608.25523v1 Announce Type: new Abstract: Prefix caching introduces a fundamental tradeoff in multi-agent large language model (LLM) serving: retaining a long system-prompt key-value (KV) cache for an agent accelerates future calls, yet it reduces the GPU memory available… 5 arXiv — NLP / Computation & Language research 3d ago Unmatched Does Not Mean False: Incomplete Reference Sets Can Reverse Calibration Rankings in Open-Ended Theory-of-Mind Tracking arXiv:2608.25654v1 Announce Type: new Abstract: Open-ended Theory-of-Mind (ToM) trackers emit valid beliefs absent from finite references. A finite-reference-plus-matcher pipeline marks unmatched outputs false, creating proxy labels that can reverse proper-score model selection… 23 arXiv — NLP / Computation & Language research 3d ago Localize-Then-Decide Guarantees for LLM Judgments arXiv:2608.25824v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used as evaluators to assess output quality and preference alignment, yet providing reliable guarantees of agreement with human judgments remains challenging. Recent work introduces… 21 arXiv — NLP / Computation & Language research 3d ago Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence arXiv:2608.25869v1 Announce Type: new Abstract: Large language models (LLMs) increasingly assess generated content, giving rise to the LLM-as-a-Judge paradigm. These systems now score outputs, filter content, and gate iterative refinement in production pipelines, where each… 21 arXiv — NLP / Computation & Language research 3d ago Retrieved But Not Reliable: A Survey on Attacks, and Defenses in Retrieval-Augmented Generation arXiv:2608.24977v1 Announce Type: cross Abstract: Retrieval-Augmented Generation (RAG) enhances large language models by grounding outputs in external knowledge, improving factuality and reducing hallucinations. At the same time, the retrieval-augmented pipeline introduces new… 32 arXiv — NLP / Computation & Language research 3d ago GGSS: Geodesic-Gated Spherical Steering for Inference-Time Debiasing of Generative Vision-Language Models arXiv:2608.25375v1 Announce Type: cross Abstract: Generative vision-language models (VLMs) are increasingly used in human-centered settings, yet they can produce demographically biased outputs even when images differ only in controlled attributes such as perceived race or… 7 r/LocalLLaMA community 3d ago this is a friendly reminder you can legally seed ai models via torrenting. A lot of people are skeptical Nvidia will keep huggingface intact now that they will buy huggingface. There's a lot of doom and gloom about not having any alternatives, removing nsfw models, saying there's no decentralized alternative or just not trusting what Nvidia might do… 38 Latent.Space news-outlet 3d ago [AINews] NVIDIA buys HuggingFace for $13B, as OpenAI publishes their HF incident retro Open Source wins! 10 The Information — AI news-outlet 3d ago Nvidia Agrees to Buy Open Source Model Repository Hugging Face For $12.9 Billion Nvidia has agreed to buy Hugging Face, a company known for its GitHub-like repository of open-source AI models, for $12.9 billion, according to a person with knowledge of the agreement. The move will put Nvidia in charge of a strategic asset in the race among open-source model… 9 r/LocalLLaMA community 3d ago Nvidia has been in talks to acquire Hugging Face for more than $13 billion - Business Insider https://www.businessinsider.com/nvidia-in-talks-to-buy-hugging-face-13-billion-dollars-2026-8 Edit: The Information is reporting that the deal is done: Nvidia Agrees to Buy Open Source Model Repository Hugging Face For $12.9 Billion (paywall):… 18 Hacker News — AI on Front Page community 3d ago Nvidia agrees to acquire Hugging Face for $13B https://www.theinformation.com/articles/nvidia-agrees-buy-op... (paywalled) https://techcrunch.com/2026/08/24/hugging-face-reportedly-in... Comments URL: https://news.ycombinator.com/item?id=49458161 Points: 247 # Comments: 113 29 OpenAI Python SDK releases dev-tools 3d ago v3.5.0 3.5.0 (2026-08-27) Features api: make function call output call IDs optional ( #3738 ) ( c74501d ) 38 Page 1 of 10 · 500 articles Older →
