News / #gpu Tag Gpu 500 articles archived under #gpu · RSS Sign in to follow r/LocalLLaMA community 8d ago Buying a V100/older NVIDIA GPU? Run this to check for older memory issues I recently bought a 32GB V100 off eBay and thought I was all set with rudimentary tests showing zero issues. But then I started seeing weird VRAM-related errors in llama.cpp. I sic'ed Claude on it, to find that ECC was disabled and that can hide small RAM issues. I believe the… 19 r/LocalLLaMA community 9d ago Does PCIe matter much for inference? Speaking of -sm tensor. I have 2x5060ti and i get some 40-50tps form 3.8 27B q6. I used HWinfo to see how saturated the PCIes are during inference are and as expected both the PCIe 5x16 slot and PCIe 4x4 were fully saturated. I cant help but feel like my 2nd gpu slot is a… 7 r/LocalLLaMA community 9d ago Qwen3.8-27B at 262K context on a Strix Halo + RTX 3090 Ti: 9.5 -> 153 tok/s, and it beats a dual-3090 vLLM box on HumanEval Spent a while treating layer placement, KV format and llama.cpp itself as experimental variables. 159 logged experiments. Numbers first, caveats after. Hardware: AMD Ryzen AI MAX+ 395 (Strix Halo, 128 GB unified) + RTX 3090 Ti on an eGPU link. One llama.cpp process, AMD on… 37 r/LocalLLaMA community 9d ago Fastest qwen 3.8 27b for AMD gpu? Hey, just wondering if there are forks or exact gguf versions that give fastest prompt processing and token gen speeds for AMD gpu? Looking to run q8 or q6 Vram 96gb W7900 + w7800 both 48gb With bandwidth mismatch, tensor paralleling amd equivalent not working   submitted by… 21 Latent.Space news-outlet 9d ago [AINews] Poolside gets $12B reverse-execuhire to NVIDIA; founders stay for $1B, employees go for $6B, Infraco scaling to 7GW neocloud Yes, we’re confused too. 28 Hugging Face Daily Papers research 9d ago FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving Abstract FlashPrefill V2 improves long-context serving via mean-corrected sparse attention, optimized GPU operators, and framework integration, achieving large speedups over dense baselines. Generated by thinkingmachines/Inkling-Small Long-context modeling is a pivotal… 9 arXiv — Machine Learning research 9d ago Finite-Horizon Input-Output Dynamics of Minibatch Perturbations in AdamW arXiv:2608.19762v1 Announce Type: new Abstract: A minibatch can influence training beyond the update at which it is observed because AdamW stores past gradient information in its optimizer states. We study this delayed effect through paired trajectories that differ only in one… 33 arXiv — Machine Learning research 9d ago Quantum Gaussian processes for prediction of channel observations arXiv:2608.19306v1 Announce Type: cross Abstract: Given a set of input states, we consider the task of predicting the expectation value of a Pauli observable at the output of an unknown quantum evolution, using only a limited number of measurements. Recently, quantum Gaussian… 32 arXiv — Machine Learning research 9d ago HiRA-CAM: Preserving Fine-Grained Spatial Relevance in Gradient-Based Visual Explanations arXiv:2608.19407v1 Announce Type: cross Abstract: Deep Learning models can include billions of parameters or more, making it difficult to explain their internal transformations and outputs. However, explainability is increasing in importance due to the use of AI in crucial… 17 arXiv — NLP / Computation & Language research 9d ago Hallucination as a Feature, not a Defect: Evaluating a multi-agent architecture to transform speculative language-model outputs into testable scientific hypotheses arXiv:2608.19206v1 Announce Type: new Abstract: Contemporary Large Language Models (LLMs) are increasingly aligned to suppress hallucinations, prioritizing factual retrieval over combinatorial creativity. While crucial for mitigating misinformation, this alignment may also… 29 arXiv — NLP / Computation & Language research 9d ago PersonalBench: Measuring the Authorship Gap in LLM Personalization arXiv:2608.19746v1 Announce Type: new Abstract: Personalized text generation aims to make LLMs write in a specific individual's style, yet existing benchmarks measure task accuracy or preference alignment rather than whether the model's output actually resembles the target… 19 arXiv — NLP / Computation & Language research 9d ago Auditing Cross-Lingual Fairness in Language Model Watermarking arXiv:2608.20047v1 Announce Type: new Abstract: Watermarking schemes for large language model output are evaluated almost exclusively on English text using each scheme's detection threshold and a narrow set of quality measurements. Multilingual deployment exposes… 9 arXiv — NLP / Computation & Language research 9d ago When Text and Numbers Disagree: Evidence Arbitration in Large Language Models arXiv:2608.20116v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used in settings where textual summaries, numerical observations, and external tool outputs may provide conflicting evidence. We study how LLMs arbitrate between such sources when they… 19 llama.cpp releases dev-tools 9d ago b10534 CUDA: adding switch points per HW and quant type to tune the mvq->MMQ decode crossover ( #26079 ) CUDA: runtime GGML_CUDA_MMVQ_MAX to tune the mvq->MMQ decode crossover Add a runtime override of the mul_mat_vec_q -> MMQ batch crossover (default MMVQ_MAX_BATCH_SIZE). Lowering it… 30 r/LocalLLaMA community 9d ago I did it! I'm free! It's been 7 hours since I used claudecode My Pro subscription expired today, they killed my access at 1pm local time. I'm now using Qwen3.8-27b w/ 5090m 24gb vram and pi to do everything i was doing in claudecode. The only downside is claudecode let me code without using my gpu, meaning I have to plan things now. Last… 15 r/LocalLLaMA community 9d ago Qwen 3.8 27b - PI AGENT vs OPENCODE https://www.reddit.com/r/LocalLLaMA/comments/1j7r47l/i_just_made_an_animation_of_a_ball_bouncing/ This post inspired me to make that test after a year ;) That is one of my many tests I make comparing output quality. What is more interesting using a PI Agent results are much… 35 Vercel — AI dev-tools 9d ago GPT-5.6 Sol is now 50% off a lower price OpenAI lowered list pricing for GPT-5.6 Sol , and the 50% AI Gateway discount now applies to the new, lower price through September 18. Input drops 20%, output drops a third. The discount applies on every OpenAI service tier: Service tier You pay now (input / output) New list… 32 r/LocalLLaMA community 9d ago A happy user of llama-swap I recently considered purchasing another GPU, not so much because I need more bandwidth or vram, but because I have several local GPU-based scheduled workloads that use different models/config. After shopping around looking at prices, I decided to use a different approach. After… 29 The Information — AI news-outlet 9d ago Nvidia to Reportedly Pay $6 Billion in Licensing and Hiring Deal with AI Model Startup Poolside Nvidia has agreed to pay $6 billion to license AI model-development software from startup Poolside, the startup told investors in a letter first reported by Newcomer . Poolside was an early developer of a coding AI agent and pivoted to developing data centers before releasing… 6 r/LocalLLaMA community 9d ago NVIDIA dropped an NVIDIA-hosted CUDA MCP for AI-assisted CUDA operations, such as searching official, up-to-date documentation, writing optimized GPU code, and analyzing performance data   submitted by   /u/swagonflyyyy [link]   [comments] 31 The Information — AI news-outlet 9d ago AT&T is Using Open Source Models to Curb Anthropic Bills Anthropic and OpenAI had better hope more companies don’t follow the example of AT&T. The telecommunications firm plans to keep its employees’ spending on Anthropic and OpenAI models flat in the coming years by using more open-source models such as Nvidia’s Nemotron, according… 5 The Information — AI news-outlet 9d ago Nvidia Plots China Comeback With New AI Chip Nvidia plans to begin small-batch shipments of an AI chip specially tailored for Chinese customers by the end of the year, according to two employees, in what would represent a new route to China for the company. Several Chinese customers have already placed orders for the chip,… 25 Hacker News — AI on Front Page community 9d ago Vomit: Clean up Claude 5's token output with a separate LLM Article URL: https://github.com/zachahn/vomit Comments URL: https://news.ycombinator.com/item?id=49375996 Points: 248 # Comments: 245 10 r/LocalLLaMA community 10d ago The boring way to run Deepseek V4 Flash-0731 130-150 tks - 16x5060ti 16GB over 2 PLX88096 switches Component Validated configuration Motherboard ASRock Rack SPC621D8U-2T/OVH CPU Xeon Gold 6330 (Get gold/platinum if interested in Optane Pmem gimmicks) GPU fabric Two Broadcom/PLX PEX88096 islands, eight GPUs per island GPUs 16 x RTX 5060 Ti 16 GB OS Ubuntu 22.04.5 LTS Kernel… 15 r/LocalLLaMA community 10d ago AirLLM - Recent Updates - with Qwen3.8-27B, Kimi-K3 too AirLLM dramatically reduces inference memory usage, letting 70B large language models run on a single 4GB GPU card — without quantization, distillation, or pruning. You can even run 405B Llama 3.1 on 8GB , DeepSeek-V3 (671B) on ~12GB , and Kimi K3 (2.8T) — the largest… 12 r/LocalLLaMA community 10d ago 3 days benchmarking most llama.cpp flags on my weird 40gb vram laptop + tb4 egpu setup. Got +70% generation, +40% prefill, 60k more context, and filed a bug in llama around MTP. What I learned. tldr: went from 16~ t/s to 27~ t/s generation. got my usable context up from 220k to the full 262k without sacrificing anything. prefill also increased from 376 to 573 command I ended up with, fwiw: llama-server -m Qwen3.8-27B-UD-Q6_K_XL.gguf -c 262144 -ngl 999 -fa on \ -ctk… 5 llama.cpp releases dev-tools 10d ago b10509 ggml: add ggml_rope_set_offset (+ metal support) ( #27120 ) add params cpu kernel metal kernel add test backend ops gate other backends ggml: (cuda) support ggml_rope_set_offset ( #27121 ) rm cuda supports_op guard, fix webgpu clang-format ggml: support ggml_rope_set_offset on… 29 arXiv — NLP / Computation & Language research 10d ago Self- and Other-Labels Induce Bidirectional Bias in LLM Judges arXiv:2608.18091v1 Announce Type: new Abstract: As LLM-as-a-judge systems become increasingly widespread, self-preference in LLMs -- the tendency to favor one's own outputs -- raises growing concerns about evaluation reliability. However, it has been studied predominantly on… 31 arXiv — NLP / Computation & Language research 10d ago Do Large Language Models Hallucinate Electric Fata Morganas? arXiv:2608.18816v1 Announce Type: new Abstract: AI hallucinations - that is, outputs which are made up, cannot be verified, or contradict the source material - are generally regarded as an engineering flaw to be dealt with. This paper contends that they also have philosophical… 19 arXiv — NLP / Computation & Language research 10d ago Test-Time Scaling in the Wild: Why Exploitation, Not Exploration, Is the Bottleneck arXiv:2608.18931v1 Announce Type: new Abstract: Test-time scaling (TTS) improves language model outputs by spending additional inference compute - generating multiple candidates, searching over partial sequences, or iteratively refining drafts. These techniques yield large gains… 25 arXiv — NLP / Computation & Language research 10d ago When Readability and Source Retention Diverge: An Evaluability Gap in AI Translation arXiv:2608.19083v1 Announce Type: cross Abstract: Readable AI output can leave an evaluability gap: even when the source is shown, an overall-quality judgment may not reflect what an output preserves. We investigated how source-text condition and output rendering relate to… 5 NVIDIA Developer Blog official-blog 10d ago Developing NVIDIA Holoscan Applications with CLI, Skills, and AI Coding Agents NVIDIA Holoscan is a platform for building real-time AI applications at the edge, from medical imaging to robotics. HoloHub is its companion repository: a... 18 NVIDIA Developer Blog official-blog 10d ago Building Federated Multimodal AI Workflows with NVIDIA FLARE Modern vision-language models (VLMs) can support tasks such as visual question answering, captioning, and image-text reasoning. In practice, however, the data... 36 The Information — AI news-outlet 10d ago Nvidia Discusses Funding Its AI Data Supplier Mercor at a $20 Billion Valuation Nvidia has discussed an investment in Mercor, a data labeling provider that helps the chip designer develop its open-source AI models, according to a person with knowledge of the process. The investment would be part of a $20 billion-valuation round. Existing investor General… 13 TechCrunch — AI news-outlet 10d ago Meet the startup helping Wall Street put a price on AI compute The AI buildout shows no signs of slowing. And with hundreds of billions of dollars a year going into data centers and GPUs, compute has become the single biggest cost for anyone building AI products. But for all that spending, there… 10 NVIDIA Developer Blog official-blog 10d ago Post-Train NVIDIA Cosmos 3 Edge for On-Device Robot Control Robots need policies that can adapt to their sensors, environments, and tasks while running on onboard computing hardware. World models offer a foundation for... 24 NVIDIA Developer Blog official-blog 10d ago Evaluating AI Agent Skill Performance with NVIDIA SkillEvaluator AI agents are only as effective as the context they receive. Even with capable models and well-documented NVIDIA libraries, agents can spend extra steps finding... 18 Hugging Face Daily Papers research 10d ago PTXBench: Benchmark and Adapt LLMs for GPU Kernel Optimization with Architecture-specific PTX Abstract PTXBench evaluates large language models on architecture-specific GPU kernel optimization, revealing uneven success and performance gaps that supervised fine-tuning only partially addresses. Generated by thinkingmachines/Inkling-Small We introduce PTXBench, a benchmark… 27 Hacker News — AI on Front Page community 10d ago Casio F-B100W-1A Article URL: https://www.casio.com/uk/watches/casio/product.F-B100W-1A/ Comments URL: https://news.ycombinator.com/item?id=49362887 Points: 326 # Comments: 263 25 r/MachineLearning community 10d ago Same effective batch does not mean same training time with gradient accumulation, tested on LoRA on T4 and L4 [D] I had assumed 1 × 4 , 2 × 2 and 4 × 1 will take somewhat similar time because effective batch is 4 in all cases. They did not. I ran Qwen3-1.7B with TRL and LoRA for 100 optimizer updates. GPU 1 × 4 2 × 2 4 × 1 T4 287.6s 258.8s 238.2s L4 213.02s 119.47s 124.76s Model, data,… 34 Hacker News — AI on Front Page community 10d ago Geolocating a random island using geometry and CUDA programming Article URL: https://yassa9.github.io/osint/gralhix-004/ Comments URL: https://news.ycombinator.com/item?id=49360545 Points: 240 # Comments: 49 24 Hugging Face Daily Papers research 11d ago EDITBRIDGE: Towards Faithful and Efficient Ultra-High-Resolution Image Editing Abstract EditBridge enables efficient ultra high-resolution image editing via a diffusion bridge that translates low-resolution edits to high-resolution outputs while preserving source details through sparse attention. Generated by thinkingmachines/Inkling-Small High-resolution… 11 Hugging Face Daily Papers research 11d ago MathForm: Scaling Mathematical Autoformalization with Knowledge Retrieval and Verification-Guided Refinement Abstract MathForm improves autoformalization by retrieving Mathlib knowledge and iteratively refining outputs with verification feedback, yielding a large verified dataset and a high-performing 8B model. Generated by thinkingmachines/Inkling-Small Autoformalization is commonly… 7 arXiv — Machine Learning research 11d ago Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Requirements arXiv:2608.17310v1 Announce Type: new Abstract: Reinforcement Learning (RL) has been promising in single-turn LLM fine-tuning. However, long-horizon agentic reasoning introduces increasingly branching interactions and sparse rewards, exposing several limitations of RL: its… 25 arXiv — Machine Learning research 11d ago rl-triton: High-Performance Triton GPU Kernels for Reinforcement Learning Credit Assignment arXiv:2608.17641v1 Announce Type: new Abstract: We present rl-triton, an open-source library of high-performance GPU kernels for reinforcement learning credit assignment, implemented in Triton. The core contribution is a unified associative scan framework that recasts seven… 38 arXiv — Machine Learning research 11d ago Cross-View Correspondence Is a Measurement Intervention: Two-Sided Validation for Agent Evaluation and Credit Assignment arXiv:2608.17713v1 Announce Type: new Abstract: Agent evaluations and trace-based learning often compare outputs across transformed views through a post-response correspondence treated as neutral preprocessing. We show that this correspondence is a measurement intervention:… 33 arXiv — Machine Learning research 11d ago A Constant-Competitive Algorithm for Dynamic Mixture-of-Experts Serving arXiv:2608.16947v1 Announce Type: cross Abstract: Huang, Lou, and Xiao introduced Dynamic Mixture-of-Experts Serving and gave an O(sqrt(log k))-competitive randomized algorithm for its integral primal problem, where k is the number of replica GPUs beyond the mandatory copy of… 22 arXiv — NLP / Computation & Language research 11d ago The Price of Thinking: Reasoning Effort as a Model-Specific API Contract arXiv:2608.16956v1 Announce Type: cross Abstract: API buyers purchase a dated contract, not a model name alone: the contract includes the requested and served model, reasoning-effort term or its omission, output rail, service product, prompt, and price schedule. We study the… 16 arXiv — Machine Learning research 11d ago Probing the Prefill: Detecting Code Vulnerabilities via Latent Activations arXiv:2608.16970v1 Announce Type: cross Abstract: LLM-based code generation is now embedded in mission-critical pipelines, but defenses against vulnerable output remain post-hoc -- static analyzers, fine-tuned classifiers, or an LLM judge that screen completed code, ignoring the… 33 arXiv — NLP / Computation & Language research 11d ago ArguLens: An Open-Source System for Automated Essay Scoring and Label-Aware Feedback Generation arXiv:2608.17356v1 Announce Type: new Abstract: Most automated essay scoring (AES) systems output a single holistic score without interpretable evidence and rely on closed APIs that introduce data privacy and cost barriers. We present ArguLens, an opensource, locally deployable… 10 Page 4 of 10 · 500 articles ← Newer Older →
