News / #gpu Tag Gpu 500 articles archived under #gpu · RSS Sign in to follow LangChain releases dev-tools 7h ago langchain-openai==1.7.2 Changes since langchain-openai==1.7.1 release(openai): 1.7.2 ( #41245 ) fix(openai): keep Responses stream items on one block when output_index drifts ( #41121 ) 23 r/LocalLLaMA community 14h ago NVIDIA's Run:AI is now apparently under Apache-2.0 license.   submitted by   /u/Ps3Dave [link]   [comments] 28 r/LocalLLaMA community 16h ago Moving a running model's layers between GPUs mid-reply (8 ms, no re-prefill) I've been working on an inference engine called jitllm. The idea came from my background in operating systems and JIT compilers: generate kernels for whatever hardware is there, and treat layers a bit like processes the engine can place and move around. In the video, a cloud VM… 35 llama.cpp releases dev-tools 20h ago b11609 webgpu: (Fix CI) fix mul_mat_id with duplicate expert ids ( #30451 ) Fix mm-id of webgpu in the case of ids duplication formatting fix comments to avoid breaking up sentences in comments Website: https://llama.app Attestations:… 4 llama.cpp releases dev-tools 22h ago b11607: chat: replace output string with parts (#30389) chat: replace output string with parts reserve first 34 r/LocalLLaMA community 22h ago Someone is running Qwen 27B at ~20 tok/s on a jailbroken PS5's GPU   submitted by   /u/LatentSpacer [link]   [comments] 33 NVIDIA Developer Blog official-blog 23h ago How to Rewrite Apache Spark UDFs for GPU Acceleration with AI Agents Enterprise applications have become increasingly demanding. Agentic AI has compounded the need for vast amounts of enterprise data with frequent, complex... 37 llama.cpp releases dev-tools 1d ago b11594 vulkan: allow multiple devices to access gpu-visible host memory ( #29741 ) vulkan: allow multiple devices to access gpu-visible host memory clean up erroneous newlines add Windows support handle nullptr fix enable condition Website: https://llama.app Attestations:… 26 llama.cpp releases dev-tools 1d ago b11603: vulkan: Optimize pool_2d when the pooling window is large (#30119) When the window is larger than a subgroup, use a whole subgroup to compute each output. 8 r/MachineLearning community 1d ago Is human evaluation through surveys common in ML research?[R][D] Hi, I am a master's student working on a project where our fine-tuned AI model provides feedback on images. We are also comparing its feedback with outputs from Qwen/OpenAI APIs on the same images. My postdoc and PhD mentors suggested creating a Google Form where participants… 36 r/LocalLLaMA community 1d ago PSA for 3090 FE owners: Running LLMs overheats your VRAM https://preview.redd.it/p6i9qifyy8vh1.png?width=2069&format=png&auto=webp&s=2b1e3c7c4fc95f12f0f7d417a4d21410aebfbd35 The 3090 Founders edition has a reputation for hot VRAM. Since LLM token generation maxes out VRAM's read capacity, I checked VRAM temps with Nvidia's stock… 37 NVIDIA Developer Blog official-blog 1d ago NVIDIA Run:ai Is Now Open Source as Part of NVIDIA DSX AI Factory Platform Today, NVIDIA is making the full NVIDIA Run:ai platform available as open source software, giving customers, developers, and ecosystem partners access to... 32 llama.cpp releases dev-tools 1d ago b11597: graph: skip the LM head for pooled embeddings (#30445) graph: skip the LM head for pooled embeddings A context that extracts pooled embeddings outputs one vector per sequence, but every model still expanded its LM head, so the graph computed n_vocab logits per token and the context kept a host buffer for them. The models now only… 31 r/LocalLLaMA community 1d ago An 80B MoE (Qwen3-Next, 48 GB GGUF) answering in a Chrome tab: llama.cpp's WebGPU backend, experts paged in from disk Follow-up to last week's post about streaming MoE experts from disk in a browser tab. That ran on hand-written engines for two models. This week the same disk tier sits under llama.cpp's own WebGPU backend, so it works for the MoE models llama.cpp already runs, with no per-model… 16 Hacker News — AI on Front Page community 1d ago A Dynamic and Distributed Virtual GPU Cloud Built on Idle Apple Silicon Devices Article URL: https://machinelearning.apple.com/research/distributed-gpu-cloud Comments URL: https://news.ycombinator.com/item?id=50070593 Points: 214 # Comments: 90 9 llama.cpp releases dev-tools 1d ago b11592: cuda : add GGML_CUDA_GRAPH_DEBUG and read env vars once (#30327) ggml-cuda: gate CUDA graph debug logs behind GGML_CUDA_GRAPH_DEBUG ggml-cuda: cache env var lookups in static const instead of re-reading per call ggml-cuda: read env vars exactly once via ggml_cuda_env_flag/ggml_cuda_env_int ggml-cuda: consolidate env flag reads into single… 21 r/LocalLLaMA community 1d ago Based on advice from LocalLLama implemented a n+2 lookahead prediction for ssd streaming Qwen Flash Next - and it worked! My project is optimising qwen flash next on Mac Mini m5 64gb through SSD streaming. I recently asked LocalLLama for guidance on how to optimise token output. Today a popular method to do this is a lookahead guesser which fetches experts one layer ahead of the current layer. One… 13 r/LocalLLaMA community 1d ago SonicBoom: 52 tok/s decode on a laptop RTX 3050 (llama.cpp: 45) with the same quantized kernels, the difference is a graph compiler. No Python. I’ve been building SonicBoom, a native tensor runtime written in C++/CUDA with a C ABI and no Python anywhere in the stack. I just tagged v0.0.1, and the first target is local LLM inference on consumer GPUs. Numbers (Gemma 4 E4B-it Q3_K_M on a laptop RTX 3050 8 GB, both runtimes… 12 llama.cpp releases dev-tools 1d ago b11586: cuda : warnings cleanup (#30420) cuda : warnings cleanup Signed-off-by: Adrien Gallouët angt@huggingface.co Update ggml/src/ggml-cuda/common.cuh Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@huggingface.co Signed-off-by: Adrien Gallouët angt@huggingface.co Co-authored-by: Sigbjørn Skjæret… 25 llama.cpp releases dev-tools 1d ago b11581 models : remove redundant ggml_cont calls now supported by all unary kernels ( #30198 ) models: remove redundant ggml_cont calls now supported by CUDA unary kernels models: restore cont_2d after sigmoid for broadcast mul, clean dflash cont Update src/models/qwen3next.cpp… 14 r/LocalLLaMA community 1d ago Anyone feel like reverse-engineering CUDA? I've been reading about all these modders using AI to reverse-engineer games and software. I've been trying out photocraft and gridcraft and have just watched a GTA5 mod featuring Sonic. It's pretty damn impressive. As a struggling AMD Radeon owner I'd love to see AI generated… 24 r/LocalLLaMA community 1d ago NVIDIA’s 9-Year-Old Tesla V100 Still Packs Some Inference Muscle; 4 GPUs On An NVLink Board Can Outperform M5 Ultra, DGX Spark In Various AI Benchmarks By Up To 230% Didn't realize the V100 were still so fast, anyone have first hand experience on these?   submitted by   /u/chemist_slime [link]   [comments] 14 r/LocalLLaMA community 1d ago Intel "Crescent Island" GPU Gets D-Series Name, Supports Up to 480 GB of LPDDR5X The newly named D-Series GPU has up to 480GB of LPDDR5X memory, 1.5TB/s of memory bandwidth Intel rates it for up to 1.3 PetaFLOPS of FP8 performance, with support for formats ranging from FP4 to FP64. It features a total of 32 Xe3P cores, 256 XMX engines, 32 MB of unified L2… 16 Hacker News — AI on Front Page community 1d ago Intel D270 GPU AI Accelerator–Up to 480 GB LPDDR5X Article URL: https://www.intel.com/content/www/us/en/products/details/discrete-gpus/data-center-gpu/d-series.html Comments URL: https://news.ycombinator.com/item?id=50063365 Points: 203 # Comments: 105 37 llama.cpp releases dev-tools 1d ago b11576 graph: pad the KV dimension of the no-cache attention to 256 ( #30416 ) The no-cache path builds a [n_tokens, n_tokens] mask and passes K and V unpadded, so FLASH_ATTN_EXT sees K->ne[1] == n_tokens. The CUDA kernels for head size 512 require a multiple of FATTN_KQ_STRIDE, so the… 37 r/LocalLLaMA community 1d ago Explore DNA with DeepMind’s AlphaGenome and in-browser MedGemma 1.5 4B via WebGPU Hey everyone, Sharing an educational tool called AskCyph Health Explorer to explore how single-letter DNA mutations lead to disease, how they alter 3D anatomy, inheritance probabilities, and available treatments. To make it interactive while protecting user privacy, the… 33 r/LocalLLaMA community 2d ago In 2030 owning a gpu cluster will be seen the same as owning a gun for home defense My window will have a sign that says "we don't call tech support" with a simplified pic of a v100 on it I know this reads like a shitpost and I'm half joking and half serious Edit: I'd sell these on Etsy if Jensen wouldn't hit me with a C&D   submitted by  … 27 NVIDIA Developer Blog official-blog 2d ago ReplaySSM: Rethinking State Storage for Faster Hybrid-Model Decoding Hybrid models interleave state-space model (SSM) layers with a few attention layers. NVIDIA Nemotron 3, Qwen3.5 through Qwen3.8, Kimi-K3, and GLM-5.3-Flash are... 26 The Information — AI news-outlet 2d ago Microsoft Debuts New Homegrown AI Chip, Plans to Deploy Multiple Gigawatts Next Year Microsoft on Monday announced the latest generation of its homegrown AI chips, dubbed the Maia 300, and said it plans to set up “multi-gigawatt deployments” of the chip starting next year. The move shows how Microsoft is aiming to reduce its dependence on Nvidia-designed GPUs,… 17 The Information — AI news-outlet 2d ago Microsoft Debuts New Homegrown AI Chip, Plans to Deploy Multiple Gigawatts Next Year Microsoft on Monday announced the latest generation of its homegrown AI chips, dubbed the Maia 300, and said it plans to set up “multi-gigawatt deployments” of the chip starting next year. The move shows how Microsoft is aiming to reduce its dependence on Nvidia-designed GPUs,… 31 r/LocalLLaMA community 2d ago graphsignal-top: a `top` for inference GPUs in the open-source Graphsignal profiler, kernels ranked by time, live, with readable names A few weeks ago we posted the open-source Graphsignal profiler here: wrap vllm serve / sglang serve / whatever in graphsignal-run and it serves everything it measures as JSON, mainly so an agent can read it in a loop. Turns out we kept opening the JSON ourselves to see what the… 20 r/LocalLLaMA community 2d ago NVIDIA RTX 5080 24GB reportedly launching in Q1 2027, board partners still waiting for details   submitted by   /u/chemist_slime [link]   [comments] 33 r/LocalLLaMA community 2d ago Qwen released QwenGyre - RL framework for training AI agents on extremely long complex tasks Introducing QwenGyre, bringing elastic RL to extreme-long-horizon (XLong) agent training. It launches with two core designs: - Elastic Scheduler: reallocates GPUs between rollout & training as demand shifts—live executions never interrupted. Up to 1.85× speedup over Colocate,… 13 r/LocalLLaMA community 2d ago Any takers for local llm? NVIDIA’s CUDA Moat Is Accelerating RTX Spark Laptop Sales, Higher 128GB Unified Memory Options Belonging To Surface Laptop Ultra Already Out Of Stock Cheapest one I see at MC is the asus pro-art version for 6499. https://www.microcenter.com/product/715662/asus-proart-p14-14-wqxga-120hz-oled-laptop   submitted by   /u/chemist_slime [link]   [comments] 33 llama.cpp releases dev-tools 2d ago b11569: metal : fix mul_mat_id with duplicate ids (#30373) metal : fix mul_mat_id limits and kernel name metal : fix mul_mat_id with duplicate ids kernel_mul_mm_id_map0 assumed at most one match per token and a fixed capacity of ne21 entries per expert, which produced wrong or dropped output rows and could overflow into the slice of the… 27 r/LocalLLaMA community 2d ago I made an open-source AI leaderboard that shows the best models your machine can run Hey everyone, I made whatmodel.sh To use it, Enter your RAM and GPU See what models you can run locally with its rank and the command to run it Compare them with API models on the same score Building it made me realise a 32GB Mac can now run a model that scores on-par with… 11 r/LocalLLaMA community 2d ago strix-llama new branch: prompt processing +21–54% over Vulkan and up to 2.7× over stock ROCm kernel at 128k, same outputs quality I've been tuning llama.cpp on my Strix Halo box (Radeon 8060S, 128 GB) and the numbers are good enough to share. It's a branch called rdna35-perf on top of halo-box/strix-llama.cpp , a llama.cpp fork aimed at Strix Halo: https://github.com/asf0/strix-llama.cpp/tree/rdna35-perf… 28 arXiv — Machine Learning research 2d ago Transformer-Based Masked Diffusion Models Provably Learn Distributions with Latent Structure arXiv:2610.12544v1 Announce Type: new Abstract: Masked diffusion models (MDMs) generate discrete data by iteratively unmasking multiple tokens in parallel using a learned mask predictor. Their output carries two errors: parallelization error due to parallel sampling, and… 7 arXiv — Machine Learning research 2d ago In-Context Learning for System Identification: Probabilistic Formulation, Long-Sequence Handling, and Benchmark Validation arXiv:2610.13411v1 Announce Type: new Abstract: Traditional system identification typically involves two steps: initially, training data is processed to estimate a model of the system of interest; then, the model is used to predict outputs for new input queries. In contrast,… 16 arXiv — NLP / Computation & Language research 2d ago AutoIndexer: Flexible-Order Decoding by Training Causal Models on Chains of Edits arXiv:2610.12545v1 Announce Type: new Abstract: Autoregressive decoding is less suited to non-causally ordered data, e.g., cross-references or future dependencies, since their outputs cannot be revised post hoc, unlike iterative human reasoning. Diffusion LLMs partially mitigate… 33 arXiv — NLP / Computation & Language research 2d ago Forecasting Corpus-Sampling Disagreement in Masked Text Reconstruction arXiv:2610.12987v1 Announce Type: new Abstract: If a second text corpus were collected from the same population, how much would a predictor's outputs change? We study this question for masked-text reconstruction using a smoothed predictor built from nearby token-pair counts and… 15 arXiv — NLP / Computation & Language research 2d ago How Reliable are AMR Parser Outputs? Uncertainty Estimation for AMR Parsing arXiv:2610.13236v1 Announce Type: new Abstract: Recent advances have significantly improved the performance of Abstract Meaning Representation (AMR) parsers, yet determining when their outputs can be trusted remains largely unexplored. This work presents the first study of… 22 arXiv — NLP / Computation & Language research 2d ago Do Prompt Optimizers Break Requirements They Are Not Scored On? A Preregistered Study arXiv:2610.13287v1 Announce Type: new Abstract: Prompt optimizers rewrite a prompt to raise one target metric, but prompts also state requirements the metric does not score, such as an output format, citations, or abstention. In a preregistered study, we run GEPA, MIPROv2, and… 15 llama.cpp releases dev-tools 3d ago b11559: opencl: improve fa, allow dk512 for gemma-4, improve dk64 (#30266) opencl: enable Gemma-4 E4B GPU decode Co-authored-by: Hongqiang Wang wangh@qti.qualcomm.com opencl: extend Gemma-4 GPU decode to E2B Co-authored-by: Hongqiang Wang wangh@qti.qualcomm.com opencl: optimize DK64 GQA8 decode Co-authored-by: Hongqiang Wang wangh@qti.qualcomm.com… 17 llama.cpp releases dev-tools 3d ago b11557: rpc : turn GGML_RPC_DEBUG into a verbosity level and add logs (#29544) rpc : turn GGML_RPC_DEBUG into a verbosity level and add logs GGML_RPC_DEBUG is now parsed as a number: 0/unset disables debug logs, 1-3 emit increasingly detailed output (events, per-command trace, transport detail). Non-numeric values fall back to 1. The duplicated env/macro… 32 r/LocalLLaMA community 3d ago Nvidia in talks to acquire US ‘open’ model start-up Reflection AI https://www.ft.com/content/052610c5-22b4-4dd4-932e-b7f9f0628b6a   submitted by   /u/External_Mood4719 [link]   [comments] 4 r/LocalLLaMA community 3d ago Here’s why RTX 5090 costs $5,000: AI firms are buying gaming GPUs by the pallet   submitted by   /u/chemist_slime [link]   [comments] 11 r/LocalLLaMA community 3d ago UPDATE: Qwen 3.8 27B 140 tok/s on single RTX 3090 Megakernel: KL divergence 0.0009 vs llama.cpp This is a follow-up to my post from yesterday ( https://www.reddit.com/r/LocalLLaMA/comments/1x2erdj/qwen3827b_on_a_single_3090_140_toks_on_code_with/ ) detailing the CUDA megakernel. Thank you all for the positive feedback and PRs! RECAP: The megakernel is a CUDA engine for… 12 r/LocalLLaMA community 3d ago Now you can grow Bonsai on your potato No more excuses for GPU-poor folks not to start LLMing! Full 27B-class reasoning in ternary transformer weights, for llama.cpp (CUDA, Metal, CPU) ~9.3x smaller than FP16 (ideal) | 98.2% of FP16 intelligence retained | ~47 tok/s on an Apple M5 Max laptop Highlights ~5.9 GB… 6 r/LocalLLaMA community 3d ago microsoft/AesCode 8B and 32B https://preview.redd.it/r2ledkyuntuh1.png?width=1684&format=png&auto=webp&s=9ed4518b2d0f20f44835ebbfac605581da186a57 AesCode generates information-rich visual artifacts such as slides, posters, and dashboards as HTML/CSS. The output remains structured, editable, and verifiable ,… 8 Page 1 of 10 · 500 articles Older →
