News / #gpu Tag Gpu 500 articles archived under #gpu · RSS Sign in to follow Ars Technica — AI news-outlet 8d ago OpenAI will watermark ChatGPT outputs by default—but only in the EU Like other solutions, it is not especially reliable, and it's easy to circumvent. 20 Simon Willison community 8d ago Introducing Mistral Large 4: Le chonk Introducing Mistral Large 4: Le chonk Mistral are back in the game. Today they're releasing a preview of Mistral Large 4, a 1 trillion parameter, 49 billion active parameter model trained on their own cluster of 3,800 NVIDIA Grace Blackwell GPUs. The preview is available via… 13 TechCrunch — AI news-outlet 8d ago AI computing startup Lambda to raise $4B ahead of planned IPO Nvidia-backed Lambda is raising up to $4 billion at a $14.5 billion pre-money valuation ahead of a planned 2027 IPO, led by Coatue and Blackstone. 33 NVIDIA Developer Blog official-blog 8d ago Scale Bitwise-Deterministic Pretraining with NVIDIA Megatron Core Bitwise determinism makes large-scale pretraining easier to debug, validate, and resume reproducibly. These benefits become especially valuable when training... 28 r/LocalLLaMA community 8d ago GPT-6.1 Sol looped "leak" hints at nested models serving architecture Hello llamas. I am posting this because I believe that, despite it being closed source models, the discussion will bring value to the local AI community. As many of you probably heard, GPT-6 Astra is speculated to be a looped transformer architecture that outputs a token after… 31 NVIDIA Developer Blog official-blog 8d ago How DOCA GPUNetIO Unifies GPU-Initiated Networking Across the NVIDIA Software Stack GPU applications increasingly need networking and data movement to behave like first-class GPU-controlled operations rather than host-driven services. When the... 9 llama.cpp releases dev-tools 8d ago b11449 ggml: fix CLAMP on non-contiguous views (CPU, CUDA) ( #29517 ) ggml: fix CLAMP on non-contiguous views (CPU, CUDA) CUDA clamped ggml_nelements values flat and ignored the view strides. CPU addressed row j as j*nb01 and ignored nb02/nb03. Both now follow the strides of dims 1..3;… 7 llama.cpp releases dev-tools 8d ago b11448 cuda: BF16/FP16 conversion to f32 chunking ( #29442 ) ggml-cuda: chunk large BF16/FP16 to F32 conversions Update ggml/src/ggml-cuda/ggml-cuda.cu Co-authored-by: Johannes Gäßler johannesg@5d6.de Update ggml/src/ggml-cuda/ggml-cuda.cu Co-authored-by: Johannes Gäßler… 12 r/LocalLLaMA community 8d ago Aplomb 1: open-weights 5.3B decision model, 1M context, text/image/video/audio in one request, #1 among 4B models on the Decision Index We released Aplomb 1 today, a 5.3B decision model with open weights. It reads up to 1M tokens of text, JSON, images, video and audio in a single request, and on our API it makes a decision on a 1M-token document in about 3 seconds, at $0.02 per 1M input tokens with free output… 36 Hacker News — AI on Front Page community 8d ago OpenTPU – An open-source AI accelerator, developed by AI Article URL: https://github.com/FeSens/openTPU Comments URL: https://news.ycombinator.com/item?id=49980715 Points: 209 # Comments: 273 4 r/LocalLLaMA community 8d ago EmbeddingGemma 2 running locally in-browser on WebGPU Model: https://huggingface.co/google/embeddinggemma-2 Demo: https://huggingface.co/spaces/webml-community/embeddinggemma-2-webgpu   submitted by   /u/xenovatech [link]   [comments] 6 NVIDIA Developer Blog official-blog 8d ago AICR v1.0: Open, stable, and verifiable GPU cluster configuration GPU-accelerated Kubernetes clusters depend on compatible versions across dozens of components, each on its own release cycle: host kernels, GPU drivers,... 17 LangChain releases dev-tools 8d ago langchain-core==1.6.7 Changes since langchain-core==1.6.6 release(core): 1.6.7 ( #41087 ) fix(core): include openai redacted_content in v1 output for bedrock converse ( #41086 ) fix(core): python 3.14 hardening around inspect.signature ( #41059 ) chore(deps): bump notebook from 7.5.7 to 7.6.3 in… 6 NVIDIA Developer Blog official-blog 8d ago Control How Your GPU Shares Work with Green Contexts GPU applications increasingly consist of multiple independent components running at the same time within a single process: a latency-sensitive operator... 17 r/LocalLLaMA community 8d ago I made a Chrome extension that filters your YouTube feed with a small local model running in the browser (WebGPU, no server) My YouTube feed was mostly songs, pranks and celebrity clips, so I built a filter that judges each video title with a small decision model running inside Chrome. Nothing is sent anywhere: no server, no API key, no account, and after one download it works offline. What it does -… 37 llama.cpp releases dev-tools 8d ago b11436: ggml-openvino: fix CI tests; fix GPU regressions. (#30037) ggml-openvino: skip unselected graph branches and support DUP Upstream #29622 adds a mixed token/embd branch to every input embedding graph through ggml_build_forward_select(). Its nodes are not flagged for compute, but the backend translated them anyway, and the DUP in that… 33 r/LocalLLaMA community 8d ago NVIDIA board member Mark Stevens sold $946.63M in NVIDIA (NVDA) shares in September NVIDIA board member Mark Stevens has executed a massive series of stock sales. SEC regulatory filings reveal that Stevens sold approximately $946.63 million worth of Nvidia (NVDA) common stock over three distinct waves. He sold ~13.3% of his total NVDA position.… 28 r/LocalLLaMA community 9d ago k_llama.cpp MoE Optimizations: Expert Residency, Hybrid CPU/GPU Execution, Q2_0 Support Finally finished my fork: https://github.com/IceFog72/ik_llama.cpp Nothing else I wanted to add/try currently works In short, it now has: Bandwidth-adaptive CPU/GPU execution Shared expert-residency ideas from https://arxiv.org/pdf/2608.16157 integrated into this fork's… 21 r/LocalLLaMA community 9d ago PewDiePie getting banned twice by OpenAI while making a local model is top-tier comedy 💀 So PewDiePie decides to fine-tune a local AI model called Ajax on his own computer. Pretty normal stuff for local model fans. To make his dataset, he uses OpenAI's API. OpenAI catches him using their outputs to train another model, flags his account for breaking their terms, and… 30 The Information — AI news-outlet 9d ago Nvidia Is Rethinking Its Revenue-Sharing Deals with Cloud Firms Nvidia is reconsidering the structure of an initiative it announced earlier this summer, the AI Compute Partnership, to provide credit support to AI cloud providers that rent out Nvidia chips, in exchange for a cut of those rental revenues, The Information reported Monday .… 25 The Information — AI news-outlet 9d ago Reflection AI Announces First Open Source Model, Beam Reflection AI, a 2-year old Nvidia-backed startup, announced its first open-weight model, adding to a collection of U.S. competitors aiming to catch up to the performance of similarly free models from China. In its announcement Monday, Reflection said it is still evaluating and… 10 r/LocalLLaMA community 9d ago We swapped AdamW's optimizer states for a Fast Fourier Transform (FFT) to cut VRAM in half. Anyone else trying non-quantization methods? Hey everyone, Like most of you, we have been fighting constant OOM errors while trying to fine-tune 8B and 70B models on consumer GPUs. The AdamW optimizer states are always the biggest bottleneck. We didn't want to rely on aggressive 8-bit quantization because we were seeing… 15 TechCrunch — AI news-outlet 9d ago Reflection debuts Beam, an open-weight AI model to rival Chinese models at lower compute cost Nvidia-backed Reflection AI unveils Beam, its first open-weight model, which it says rivals GLM-5.2 on reasoning with far less inference compute. Weights are due this month. 27 llama.cpp releases dev-tools 9d ago b11430: hexagon: matmul and flash-atten scalability updates (#29974) hexagon: head-parallel flash_attn partitioning for row-split multicore In row-split mode each core computes its output row shard of every MUL_MAT, but flash_attn was previously partitioning by Q tokens (flat qrow split) instead of by heads. This forced every core to read the… 37 r/MachineLearning community 9d ago I have trained a model to predict my blood sugar (Part 2) [P] This is related to my previous post where I shared an encoder-only transformer model trained on ohiot1dm + shanghait1dm + azt1d datasets. This time I trained the model on the outputs of my T1DM patient simulator and then measured its zero-shot performance on my real-world blood… 24 llama.cpp releases dev-tools 9d ago b11425: CUDA: make the alloc_deps check batch independent (#29986) Fixes #29980 20 llama.cpp releases dev-tools 9d ago b11422: cuda: use the vector lightning indexer kernel on MUSA (#29990) cuda: stage the lightning indexer queries in head passes for MUSA MUSA archs 21 and 22 cap static shared memory at 28 KB, and the tile kernel staged the queries of all four heads next to the key tile for 33 KB. The queries are now staged in passes of… 10 The Information — AI news-outlet 9d ago How Jensen Huang’s Son-in-Law Became One of His Most Trusted Executives A little over a week ago, Madison Huang, a rising star at Nvidia and the daughter of its CEO, celebrated her wedding in Hawaii. Her husband? Nico Caprez, a former pro ski racer whom Madison helped recruit to Nvidia two and a half years ago. And he’s not just any employee.… 16 llama.cpp releases dev-tools 9d ago b11417: CUDA: Optimize accumulation in mmq for NVFP4 type (#29857) ggml_cuda: optimize accumulation in mmq_vec_dot_fp4_fp4_mma for better performance remove whitespace fix: correct indentation in mma_block_scaled_fp4 loop 36 r/LocalLLaMA community 9d ago MoE SSD streaming on a 64 GB Mac mini: GPU still waits 27% of decode on experts. Ideas? Looking for some assistance /ideation. I am running qwen flash next q4 in my Mac mini m5 64gb. QFN doesn’t fit so this is done by having as many experts hot in cache as possible and streaming in the rest from ssd. I’m getting 17.5 tks decode and 390 tks pp. Have done a bunch of… 32 r/LocalLLaMA community 9d ago Lessons learned while building Apex-2 Hi everyone, thank you so much for all the interest in my model. It's more than I expected. Here is a short summary of the trial and error I went through while building Apex-2. 1. GPUs were always the bottleneck I planned to train on about 1T tokens, but in the end I could only… 35 r/LocalLLaMA community 9d ago Ok how to actually learn vLLM ? Said in title, I find the ecosystem difficult to understand, and RTFMing doesn't help me as it's never clear what is the server vs their client library ? I'm using it for voxtral 3B on one GPU, but it's because I can run that with no quantization, I'm lost on learning to run… 18 llama.cpp releases dev-tools 9d ago b11406 webgpu: add MMVQ support for Q1_0/Q5_0/Q5_1/Q3_K/Q5_K/Q6_K/MXFP4 ( #29483 ) add supports for q1/q5/q3_k/q5_k/q6_k/mxfp4 of mmvq path Add K_QUANTS_HANDLING macro to q1_0 of mmvq path Website: https://llama.app Attestations:… 5 llama.cpp releases dev-tools 9d ago b11405 cuda: tile the lightning indexer over keys and tokens for 4 heads ( #2 … 26 llama.cpp releases dev-tools 9d ago b11403: CUDA: use MMVF for thin f16/bf16 mul_mat at small batch size (#29633) CUDA: use MMVF for thin f16/bf16 mul_mat at small batch size Signed-off-by: ynankani ynankani@nvidia.com adjust kernel selection logic Signed-off-by: ynankani ynankani@nvidia.com Co-authored-by: Johannes Gäßler johannesg@5d6.de 12 arXiv — Machine Learning research 9d ago Latent-MOPD: Latent Multi-Teacher On-Policy Distillation arXiv:2610.02381v1 Announce Type: new Abstract: On-policy distillation (OPD) trains a student on the responses it generates. Existing LLM multi-teacher OPD transfers what specialists predict through their output distributions. We introduce Latent-MOPD, to our knowledge the first… 6 arXiv — Machine Learning research 9d ago AI-driven Thermal-aware Data Center Capacity Planning arXiv:2610.02442v1 Announce Type: new Abstract: The emerging of large language models (LLMs) has posed significant challenges to the thermal management of data center. Intense GPU computation for LLMs results in localized hotspots. Moreover, spiking thermal loads during training… 6 arXiv — Machine Learning research 9d ago Activation Sparsity with Weight Approximation for Faster LLM Decoding on Offloaded Weights arXiv:2610.02598v1 Announce Type: new Abstract: Deploying LLMs on consumer-grade GPUs with insufficient memory to hold their weights can result in prohibitively slow inference, because decoding repeatedly transfers offloaded weights from system RAM or flash storage into GPU at… 32 arXiv — Machine Learning research 9d ago Online Verification of Language Model Responses Under Cost Constraints arXiv:2610.02632v1 Announce Type: new Abstract: As large language models are increasingly deployed for multi-step reasoning, verifying the correctness of their outputs has become essential for maintaining reliability at scale. Verifying the correctness of large language model… 28 arXiv — NLP / Computation & Language research 9d ago Output Language Confusion under Multilingual Prompt Contamination arXiv:2610.02926v1 Announce Type: new Abstract: Standard factual benchmarks assume clean monolingual prompts and exact-match scoring, two assumptions that break simultaneously in real-world multilingual deployment, from retrieval-augmented generation pipelines returning… 36 arXiv — NLP / Computation & Language research 9d ago Right Order, Wrong Scale: Auditing LLM Judges for Occupational AI Measurement arXiv:2610.02492v1 Announce Type: cross Abstract: LLM judges are increasingly used to assess whether AI outputs meet workplace requirements, but agreement on response rankings does not establish agreement on acceptance rates or occupational aggregates. We introduce O*NET-BENCH,… 22 r/LocalLLaMA community 9d ago Infermeld: a Linux kit for running one GGUF across AMD + NVIDIA GPUs with llama.cpp Following on from club-5060ti and club-rdna16 , I’ve put together Infermeld : a small, open-source Linux companion kit for running one GGUF across an AMD GPU and an NVIDIA GPU, powered by llama.cpp. I’m the maintainer. This is an experimental v0.1.0 release , and I’m looking for… 24 llama.cpp releases dev-tools 9d ago b11402 CUDA: prefer whole-tile FlashAttention scheduling for efficient two-s… 23 Vercel — AI dev-tools 9d ago OpenAI Ultrafast mode now available on AI Gateway AI Gateway now supports OpenAI's Ultrafast service tier for GPT-6 Astra, providing faster output for interactive applications and rapid coding iterations. To use Ultrafast, request it for openai/gpt-6-astra through AI SDK, the Chat Completions API, or Responses API: For… 28 llama.cpp releases dev-tools 10d ago b11399: CUDA: refactor swizzling code (#29612) CUDA: refactor swizzling code fix templates/loop bounds 26 r/LocalLLaMA community 10d ago Poor People Vulkan GPUs list Help with this list. Give me your recommendation on "not supported anymore" GPUs. Looking for budget and Vulkan friendly options. Most of the GPU are not supported by latest CUDA / ROCm. Often with some witchcraft magic they are able to run with native backend. I prefer the… 25 llama.cpp releases dev-tools 10d ago b11397 cuda : move neu_padded to where it is used ( #29940 ) Signed-off-by: Adrien Gallouët angt@huggingface.co Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52636582 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64,… 37 r/LocalLLaMA community 10d ago LLM Inference Dashboard Working on a resource dashboard, rich logs, lightweight 64mb cap, all local, scales on network API endpoints via collector, supports multiple engines (llama, strata, custom cuda engines, unsloth, LMS) Not public yet, but curious if anyone would be interested? It’s better logging… 7 r/LocalLLaMA community 10d ago poorman inference engine for 16GB GPU and 35B moe Qwen 3.6for coding I forked llama.cpp's server into AgrillaMoE, a dedicated build for Qwen3.6-35B-A3B (~A4B) with Unsloth quants. On a (vant.ai) rented V100 16GB with the 2-bit UD-Q2_K_XL quant it generates at ~57-60 tok/s while running the full MoE-expansion profile — and it speaks both the… 7 r/LocalLLaMA community 10d ago A Strata fork for IBM AC922 running Qwen3.8-FN UD-Q4_K_XL is doing up to 7,357 tk/s prefill and 113 tk/s decode I forked Strata and worked with Claude code with some heavy changes to it to make it work on an IBM AC922 I have access to. The IBM AC922 is a 2018 era beast with two POWER9 20 core SMT4 CPUs that are connected by NVLink to 4 or 6 NVIDIA Tesla V100 SXM2 GPUs, the CPU-GPU BW… 37 Page 4 of 10 · 500 articles ← Newer Older →
