News / #gpu Tag Gpu 500 articles archived under #gpu · RSS Sign in to follow arXiv — Machine Learning research 25d ago Provably Learning Multi-Head Attention with Queries arXiv:2608.03294v1 Announce Type: new Abstract: We study the problem of learning multi-head softmax attention from black-box input-output access. The learner may query arbitrary real-valued token sequences and observe only the scalar output at the final token. Recent work gives… 6 arXiv — NLP / Computation & Language research 25d ago Stuck on "A": Diagnosing and Repairing Interface Injury in Attention-to-KDA Linearization of a 0.6B Language Model arXiv:2608.02689v1 Announce Type: new Abstract: We convert 21 of 28 full-attention layers of Qwen3-0.6B-Base into KDA (Kimi Delta Attention) linear-attention layers on a single consumer-grade GPU budget, and ask a simple question: what exactly does the conversion break? After… 38 arXiv — NLP / Computation & Language research 25d ago ARCHead: Activation-Metric Residual Correction for Large Language Model Output Heads arXiv:2608.02703v1 Announce Type: new Abstract: Weight-only quantization substantially reduces the storage of large language model (LLM) transformer blocks, but practical backends often retain the final language-modeling head (LM-head) in BF16 or FP16. Quantizing this projection… 21 arXiv — NLP / Computation & Language research 25d ago VetScore: Risk-Weighted Fact Verification for Veterinary Long-Form QA with Citations arXiv:2608.03675v1 Announce Type: new Abstract: Citation excerpts can be used to increase the reliability of generated outputs and their faithfulness to cited sources, which is especially important in high-stakes domains such as human and veterinary medicine. However, this does… 29 arXiv — NLP / Computation & Language research 25d ago VIBE: A VAD-Informed Benchmark for Entity-Centered Affective Profiling of Large Language Model Outputs arXiv:2608.03810v1 Announce Type: new Abstract: Large language models routinely describe socially salient targets, including political figures, countries, religions, organizations, historical events, and social groups, encoding affective framing alongside factual content: a… 11 r/LocalLLaMA community 25d ago PSA Update CUDA from 13.2 to 13.3 to solve DeepSeek V4 Flash 0731 Looping Problem! So one of yall mentioned that cuda 13.1 or 13.2 is broken for unsloth so I looked in to it, and they were right. I had 13.2 installed, after I switched to 13.3 no more looping!!! Before the cuda update, the model was literally unusable. A few minutes into the run it would start… 20 r/LocalLLaMA community 25d ago DeepSeek-V4-Flash on SM89 4x48gb 4090s with DSpark https://github.com/yhfgyyf/vllm-deepseek-v4-sm89 I couldn't believe that someone actually got vLLM working with this particular set of GPUs, but here it is. The video is from right after I got it working with 64k context, but it is now running with 256k.   submitted by  … 26 r/MachineLearning community 25d ago I Compressed Bad Apple into a 3MB Neural Network [P] I trained a small MLP to memorize the classic Bad Apple animation, ~2.7 billion pixels of video compressed into 790k parameters (3.2 MB float32, 1.6 MB float16). The network takes a 3D coordinate (t, y, x)- frame index and pixel position- and outputs a grayscale value between 0… 12 llama.cpp releases dev-tools 25d ago b10275 server: decode Windows OEM output to UTF-8 in built-in tools ( #26597 ) a child process writes in the OEM code page, which is not UTF-8 on a western Windows install, so accented output reaches the JSON layer as invalid bytes and gets replaced there, silently losing the… 38 TechCrunch — AI news-outlet 25d ago Nvidia doesn’t mess around: A week after open AI industry group formed, it’s already showing progress The week-old Open Secure AI Alliance, spearheaded by Nvidia and grown to over 120 companies, already has proposals out for defending against AI agents. 5 r/LocalLLaMA community 26d ago Hugging Face CEO says China is winning the AI race and dominating on open models This is something that was spoken here and there, and now it is like writing on the wall. The main additional point is that China has created an independent supply chain. Starting from raw materials and home-made lithography equipment, through their own GPU manufacturing, and to… 28 r/LocalLLaMA community 26d ago A llama.cpp PR caches “hot” MoE experts on the GPU — 33 → 56 tok/s reported with 8GB VRAM A new llama.cpp PR (#26563) adds a heatmap that tracks which MoE experts are used most often. Instead of keeping every expert on the GPU or offloading all of them, it caches the frequently selected experts in VRAM while the cold experts continue running on the CPU. The author’s… 29 r/LocalLLaMA community 26d ago Decrease the power limit of your 5090 to at least 480W - the performance penalty for inference is negligible. I run my inference machine in the living room, so noise and heat output are a significant concern. Ran a quick test using my daily driver model (Qwen 3.6-27b) and at 480W, the card outputs only 2.1% less t/s in decode and 8.8% in prefill (which is already very fast). Well worth… 6 NVIDIA Developer Blog official-blog 26d ago Generate Trajectories, Reasoning Traces, and Auto-Labels with NVIDIA Alpamayo 2 Super Autonomous vehicle (AV) development often relies on separate models for trajectory generation, high-level intent prediction, scene understanding, and data... 10 r/LocalLLaMA community 26d ago Llama.cpp PR 8% speed boost Llama.cpp currently uses cpu based sampling for user with mtp enabled. The PR moves sampling to the gpu, which on a 5090 boasts an 8% increase in tok/s for qwen3.6:35b. I tested it on my P40 and observed a 4% increase inference speed boost. Pretty exciting to see 84 tok/s max on… 19 r/LocalLLaMA community 26d ago Optimised DSv4-Flash for 2x GH200: 10,000 tok/s PP, >300 tok/s TG on SGLang There are some PRs to use and a nice trick to speed up PP on really longs contexts in my write up. Hope it helps! TL;DR: On this dual GH200 box, you build vLLM v0.26.0 from source, add the merged DSV4 cache-layout patch (PR #48993), disable async scheduling, and run DSpark at 6… 25 Hacker News — AI on Front Page community 26d ago DeepSeek V4 Flash on a Single AMD MI300X Article URL: https://github.com/ryanzhou/deepseek-v4-flash-mi300x Comments URL: https://news.ycombinator.com/item?id=49166386 Points: 290 # Comments: 64 32 Hugging Face Daily Papers research 26d ago GradCuit: Credit-Assigned Gradient Flow Enables Robust and Interpretable Test-Time Latent Reasoning Abstract Optimization-based latent reasoning improves large language model outputs by optimizing instance-specific continuous states at test time while keeping model parameters frozen. Existing methods, however, typically connect these states to the reasoning trajectory through… 8 Smol AI News news-outlet 26d ago not much happened today **Alibaba** launched **Qwen3.8-Max**, enhancing multimodal capabilities and agent ecosystem integration. **NVIDIA** introduced **Alpamayo 2 Super** for autonomous vehicle reasoning, while **Mistral AI** released **Shieldstral**, a 3B parameter open-weights safety model for… 17 arXiv — Machine Learning research 26d ago Caliber: Cross-Architecture Extraction-Cost Control for Score-Returning APIs arXiv:2608.01023v1 Announce Type: new Abstract: We present Caliber, an output-perturbation defense against model extraction that formulates noise selection as a calibration problem: how much the defense degrades the supervision signal used to train a surrogate, and the provable… 6 arXiv — Machine Learning research 26d ago Meganeura: Portable GPU Training and Inference through Vulkan and Metal arXiv:2608.01563v1 Announce Type: new Abstract: Training and deployed inference often cross export, conversion, and platform-specific runtime boundaries. Meganeura asks whether one compact native compiler can span both phases on consumer GPUs. Its typed static graph, automatic… 29 arXiv — NLP / Computation & Language research 26d ago Writing-System-Level Tokenizer Adaptation for Byte-Level BPE arXiv:2608.00582v1 Announce Type: new Abstract: Pretrained byte-level BPE tokenizers can segment underrepresented languages inefficiently. Replacing a tokenizer changes the meaning of nearly every token ID, while vocabulary expansion enlarges the model's embedding and output… 6 arXiv — NLP / Computation & Language research 26d ago TreeProbe : A Tibetan Medicine Benchmark for Cultural Bias in LLMs arXiv:2608.00640v1 Announce Type: new Abstract: Large language models are increasingly viewed as a potential means of mitigating global health inequities, yet their outputs often reflect dominant high-resource medical traditions and provide limited coverage of traditional… 38 arXiv — NLP / Computation & Language research 26d ago Understanding Sparse Attention Selectivity in Long-Context Foundation Models via Counterfactual Evaluation arXiv:2608.01676v1 Announce Type: new Abstract: Sparse attention is widely deployed in long-context serving stacks, yet no framework audits how discarding blocks changes the influence of specific content on model output. We first establish that the phenomenon is real and causal:… 18 Simon Willison community 26d ago Don't be a meat proxy Don't be a meat proxy Niklas Gruhn coins an excellent new term - meat proxy - for people who blindly copy and paste the output of AI systems to their peers. By all means, prompt AI. But don't just relay the output. Read it, understand it, validate it, and then write a… 17 r/LocalLLaMA community 26d ago NousResearch keeps doing things on hermes Has anyone followed nousresearch work on Hermes? I mean we are Q3 2026. We have some crazy models trickling down from HGX territory to multi gpu workstation. And we have nousresearch deploying the 0.20 of its hermes agent while starting releasing the project with a 0.2 mid… 13 r/LocalLLaMA community 26d ago nvidia/NVIDIA-NemotronLabs-VoiceChat-11B · Hugging Face (full duplex)   submitted by   /u/adefa [link]   [comments] 9 r/LocalLLaMA community 27d ago Thermal paste PSA for old GPUs I know many of us are using older GPUs like the 3090 because they work great. I just replaced the thermal paste and am seeing consistently 10 C lower temperatures. The old paste was cracking and like dry dust when I removed it. This made the difference between super loud fans… 8 llama.cpp releases dev-tools 27d ago b10242 CUDA: Add backend sampler for penalties sampler ( #25262 ) sampling: enhance penalty handling in common_sampler_init Set default value for penalty_last_n based on model context if not specified. Ensure penalty_last_n and n_prev are non-negative. Update llama_sampler_penalties… 20 r/MachineLearning community 27d ago It's time to desk reject papers that don't include code that can reproduce the results [D] As review season for NeurIPS wraps up, I have now reviewed for 3 major conferences this year. And I'm noticing a worrying trend: Out of the 12 papers I reviewed this year, only 1 provided full code (that runs the whole training pipeline from input dataset to output AUROC). 4… 28 NVIDIA Developer Blog official-blog 27d ago How to Run Isolated Tenant Kubernetes Clusters on Shared GPU Infrastructure Running a dedicated Kubernetes cluster per team often results in more isolation than an organization requires. While one cluster can be successfully shared... 9 NVIDIA Developer Blog official-blog 27d ago NVIDIA Vera Storage Benchmarks: Faster Encryption, Compression, Integrity Checking, and Recovery for AI-Native Storage Storage is an active part of every agentic AI workflow. As agents retrieve enterprise knowledge, access persistent memory, reuse key-value (KV) cache data,... 17 r/LocalLLaMA community 27d ago All DeepSeek model oneshots: 242 outputs to look at and compare! Continuing my weekend of oneshotting the cheap OpenRouter models, here are all 10 DeepSeek models across the same 35 prompts. DeepSeek had a rougher time (more provider errors / empty completions), so only 242 made it out of the 10*35 matrix. Here they are… 31 llama.cpp releases dev-tools 27d ago b10241 CUDA: Fix data-races when reusing SMEM in block_reduce ( #26385 ) CUDA: Fix data-races when reusing block_reduce block_reduce currently doesn't resync after reading from SMEM, causing potential data-races when reusing SMEM for multiple reductions. One may consider simply always… 24 r/LocalLLaMA community 27d ago I compared MinerU, Granite-Docling, and PaddleOCR-VL on 12 PDF-parsing capabilities using 6 document types I tested them by sending the 6 documents, each meant to represent a different document type, through my own webapp and comparing every output against the source. All ran on the same L4 GPU. The documents: Financial statements with merged multi-level headers (A typical annual… 37 Hacker News — AI on Front Page community 27d ago AirLLM 70B inference with single 4GB GPU Article URL: https://github.com/lyogavin/airllm Comments URL: https://news.ycombinator.com/item?id=49154228 Points: 203 # Comments: 76 20 Hugging Face Daily Papers research 27d ago ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction Abstract Enterprise workflows increasingly rely on agents for schema-guided extraction: given a document and a user-defined schema, the agent faithfully follows the schema to produce the correct output with source evidence as grounding metadata. We present ExtractBench, a… 4 r/LocalLLaMA community 27d ago Seedance 2.5 Vs Minimax H3 (Open Weight). Excellent Output Comparison!   submitted by   /u/Hannibalj2ca [link]   [comments] 23 arXiv — Machine Learning research 27d ago Topology-Aware Data Movement for Disaggregated GPU Inference arXiv:2607.28633v1 Announce Type: new Abstract: Disaggregated LLM inference creates a datacenter networking problem that no existing system solves correctly. When prefill and decode run on separate GPU pools, the KV cache must be transferred between them. For a 70B model this is… 6 arXiv — Machine Learning research 27d ago Autonomous Repair for Multi-Agent Systems via Monte-Carlo Tree Search arXiv:2607.29055v1 Announce Type: new Abstract: Multi-agent systems (MAS) are increasingly deployed to solve complex tasks. In case of incorrect or unsatisfactory outputs, users have to manually locate agent mistakes by inspecting agent trajectories (i.e., {\em failure… 9 arXiv — NLP / Computation & Language research 27d ago How Hard Does It Think? Analyzing Step-Aware Reasoning Energy in LLM Chain-of-Thought Trajectories arXiv:2607.28674v1 Announce Type: cross Abstract: Understanding how computational effort is allocated across individual chain-of-thought (CoT) reasoning steps remains an open challenge: existing interpretability methods rely on output-level signals or collapse processing depth… 28 arXiv — Machine Learning research 27d ago DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs arXiv:2607.28848v1 Announce Type: cross Abstract: LLM serving systems are provisioned for peak load to meet strict latency targets, leaving substantial GPU compute idle whenever traffic falls below peak. We present DeltaServe, a host-agnostic co-serving design that converts this… 20 arXiv — NLP / Computation & Language research 27d ago CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation arXiv:2607.29252v1 Announce Type: new Abstract: Reliable evaluation of open-ended LLM outputs requires fine-grained rubrics, yet expert curation is costly and difficult to scale. Existing automated pipelines rely on strict judge unanimity and binary variance filters, which… 19 r/LocalLLaMA community 27d ago China’s DFSX Offers 2x The Memory Bandwidth Of NVIDIA’s GB200   submitted by   /u/MundanePercentage674 [link]   [comments] 8 r/LocalLLaMA community 27d ago DeepSeek V4 @ IQ3XXS on M1 Ultra 128GB- 16 tok/s in LM Studio after patch M1 Ultra 128GB, Unsloth UD-IQ3_XXS, wired limit at 120GB. I was at 5-6 tok/s before the patch. Getting 15-16 tok/s now with the patched engine, and the output seems to have improved. Big thanks to this guy.   submitted by   /u/mil_phickelson [link]   [comments] 9 r/LocalLLaMA community 28d ago DSpark Benchmark Result on Deepseek v4 Flash 0731 TensorSharp supports DSpark on Deepseek v4 Flash 0731 now. Here is the benchmark result on 4x Nvidia A40 GPUs, cuda 12.8 with/without DSpark: Model: DeepSeek-V4-Flash-0731-UD-Q8_K_XL from https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF DSpark draft model from:… 31 r/LocalLLaMA community 28d ago All Qwen model oneshots: 1109 outputs to look at and compare! I've been busy this weekend generating oneshots for all the cheapest models on the openrouter and ended up going through all 33 qwen models across 35 prompts (there were some failures and only 1109 made out of 33*35 matrix). Here they are https://oneshotlm.com/model/?q=qwen Qwen… 35 r/LocalLLaMA community 28d ago Are you ready for Le Chaton FAT or still wasting money on GPUs? According to rumors (spread by myself) Le Chaton FAT will be 26T-a3b and I AM READY for it. Let's be real, I can't afford that many 5060Ti, so I got 12x Gen 4 3.2 TB (two per card). This gives me about 60GBs bandwidth on 30TB. Added 256gb ddr4 just for kv cache, but I can also… 8 r/LocalLLaMA community 28d ago Deepseek v4 flash - 100-150 faster t/s in prefill/pp. You have two choices here (in order of pref): Downgrade CUDA from 13.3 to 13.1 (skip 13.2 due to bugs) <- prefer this (thanks to u/fairydreaming for pointing this out) Use this vibed fork that works with CUDA 13.3 https://github.com/vektorprime/working_ds4_speed I was… 34 r/LocalLLaMA community 28d ago Deepseek-V4-Flash-0731 Dwarfstar on Mac Here is the prefill performance in an M2 Ultra with 192GB of RAM. For decode, at the following depth: Start: 28 t/s 45k: 23.5 t/s 192k: 18 t/s That speed is maintained with 8k token output at those depths.   submitted by   /u/Badger-Purple [link]   [comments] 17 Page 9 of 10 · 500 articles ← Newer Older →
