News / #gpu Tag Gpu 500 articles archived under #gpu · RSS Sign in to follow llama.cpp releases dev-tools 25d ago b11047 cuda : fix CUB argsort corruption caused by in-place keys ( #28389 ) argsort_f32_i32_cuda_cub called the one-shot DeviceRadixSort::SortPairs API with d_keys_in == d_keys_out (temp_keys, temp_keys). CUB's internal double-buffer ping-pong requires distinct key buffers: with… 22 r/LocalLLaMA community 26d ago Tuning Qwen 3.8 27B and OMP as a coding agent on 2× 3090s Oh My Pi + vLLM on two 3090s. Average wait per turn went from 28s to 7s, mostly from changing omp settings: explicit effort level on every role (unset ones defaulted to xhigh) thinking_token_budget of 7500 maxTokens 8k → 32k (file writes were getting cut off) tool output over 10… 23 r/MachineLearning community 26d ago How is RLCD (jev) RL? [D] Just saw the YouTube presentation and I was left wondering this question. If jev only outputs Choice, Score, or Noul … well those are all perfectly differentiable. (Cross entropy or mse) I don’t know if I’m missing something or if adding RL is just for marketing. Like what would… 34 The Information — AI news-outlet 26d ago Nscale IPO Files to Go Public, Shows Huge Revenue Jump And Steep Losses Nvidia-backed startup Nscale disclosed Friday that revenue had surged 10 times in the first half of this year from the year-ago period, but losses mounted as the cloud startup spent billions on data centers and the AI chips they’ll house. The two-year-old UK startup, a spinout… 12 r/LocalLLaMA community 26d ago Built this yesterday with Qwen3.8-Flash-Next (NVFP4, 262K context) on a single NVIDIA DGX Spark Planning, coding, testing = 8h total. Stack: VSCode Copilot in autopilot mode + SGLang Stats: ∼10k lines generated, ∼800k tokens consumed Sure, it's not GPT-6 Astra level, but for a 100% local ∼180B MoE running on a single DGX Spark at ∼35 tok/s. Not bad...   submitted by… 20 TechCrunch — AI news-outlet 26d ago Open or closed AI? Nvidia’s Nader Khalil and Sydney Sykes take on one of the decisions shaping next-gen startups at TechCrunch Disrupt 2026 Nvidia's Nader Khalil and Sydney Sykes discuss one of the decisions shaping next-gen startups on the Builders Stage at TechCrunch Disrupt 2026. 37 llama.cpp releases dev-tools 26d ago b11037 ggml-webgpu: fix supports_op condition for GET_ROWS ( #28978 ) fix get_rows vec4 handling Add src strides checking to vec4_aligned of get_rows and the new test case. Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48438627 macOS/iOS:… 31 r/LocalLLaMA community 26d ago Flyweight: open-source C++/CUDA engine for running MoE models bigger than your VRAM on one GPU + system RAM. First PyPI release, looking for contributors. Been building this for a few months, mostly for myself, and it just got a proper release so figured I'd post it. It's a native GGUF inference runtime with OpenAI/Anthropic-compatible APIs and a chat UI. The whole point is one consumer NVIDIA card + lots of RAM: MoE models that… 21 arXiv — Machine Learning research 27d ago Compressed Active Subspaces for Scalable Bayesian Inference arXiv:2609.19539v1 Announce Type: new Abstract: Active subspace methods provide a framework for quantifying predictive uncertainty in high-dimensional models by identifying and performing inference along parameter directions that have the greatest influence on the model output.… 5 arXiv — Machine Learning research 27d ago The Complexity Kink: A Prompt-Side Structural Complexity Index for Code-Generation Reliability arXiv:2609.19616v1 Announce Type: new Abstract: Complexity measured from generated code is failure-dependent: a difficult prompt can yield a short failing program and be assigned low output complexity. We introduce a six-dimension prompt-side structural-complexity index scored… 14 arXiv — Machine Learning research 27d ago How Far Can Sub-3B Open Language Models Go in Zero-Shot Essay Scoring on an 8 GB Consumer GPU? arXiv:2609.20250v1 Announce Type: new Abstract: Zero-shot essay scoring with large language models is usually demonstrated with proprietary API models, yet the settings where automated scoring is most needed, such as public schools grading thousands of essays under strict… 5 arXiv — NLP / Computation & Language research 27d ago Subliminal Prompting Beyond Static Geometry: Causal Depth and Multi-Token Confounds arXiv:2609.19149v1 Announce Type: new Abstract: Subliminal learning shows that language models can transmit a hidden trait through outputs that appear unrelated to it. One proposed explanation, token entanglement, links animal and number tokens through the model's output… 16 r/LocalLLaMA community 27d ago Installing 6 GPUs in a standard case rather than using an open-frame chassis. CPU : epyc 7262 MB : ROMED8-2T GPU : V100 16GB PCIE x6 I have built a server with six V100 GPUs. I am now testing it and plan to eventually run Qwen3.8-Next-Flash configured with TP2 and PP3. Because open-frame or server-style cases are large and unattractive, I chose to install… 8 r/LocalLLaMA community 27d ago Ternary Bonsai 2 (27B) just released on Hugging Face. At <6GB in size, it can even run locally in-browser on WebGPU. The model is derived from Qwen3.8-27B, a 27B hybrid-attention causal language model (architecture unchanged), but uses ternary weights to shrink model size down to <6GB in size. According to the model card, it's 9x smaller than FP16 while retaining 98.2% of the intelligence. -… 7 Hacker News — AI on Front Page community 27d ago Bend – A language that blocks AI mistakes via proof, on CPU and GPU Article URL: https://bend-lang.com/ Comments URL: https://news.ycombinator.com/item?id=49746163 Points: 250 # Comments: 132 19 r/LocalLLaMA community 27d ago AMD Plans 10% Price Hike Across GPUs, Chipsets, and Possibly CPUs Great news! AMD is also considering accepting payment in organs! Slightly less sarcastically, grab what you can, while you can. Waiting is becoming very costly almost by the day   submitted by   /u/FullstackSensei [link]   [comments] 27 TechCrunch — AI news-outlet 27d ago Huawei plans Q1 2027 launch of new AI chip as it takes on Nvidia Huawei is accelerating the launch of its next-generation Ascend 960DT AI chip as it pushes to compete with Nvidia and close China’s AI computing gap with the U.S. 38 TechCrunch — AI news-outlet 27d ago Google, Nvidia and Anthropic want Emerald AI to find space on the grid for more data centers A new coalition that includes Google, Nvidia, Anthropic and Emerald AI wants to find 100 GW of grid capacity for new data centers. 22 r/LocalLLaMA community 27d ago We built an open-source GPU profiler you point an AI agent at, instead of reading traces yourself We've been tuning vLLM/SGLang/llama.cpp setups for a long time and got tired of the profiling part: nsys trace, open the GUI, squint, change a flag, repeat. The profilers assume a human is looking at the timeline. These days the thing doing our tuning is usually an agent, and it… 26 r/LocalLLaMA community 27d ago China's Huawei says AI chip demand outstrips supply as it steps up Nvidia challenge   submitted by   /u/sunychoudhary [link]   [comments] 11 r/LocalLLaMA community 27d ago Intel releases OpenVINO 2026.4 UPDATE https://github.com/ggml-org/llama.cpp/pull/29009 MERGED More Gen AI coverage and frameworks integrations to minimize code changes New models supported: On CPU: Gemma-3n On CPU, GPU: Kokoro-82M, Qwen3-VL-4B with eagle3, Qwen3-ASR, Muse Glimmer 30B, Qwen 3.8 27B, Gemma4… 22 The Information — AI news-outlet 27d ago Huawei Speeds Up AI Chip Launch to Challenge Nvidia Huawei Technologies plans to launch its new AI chip in the first quarter of 2027, nine months earlier than planned, as it steps up its efforts to challenge Nvidia. Tao Wang, Huawei’s deputy chairman and rotating chairman, said at the Huawei Connect conference in Shanghai on… 15 Hugging Face Daily Papers research 27d ago Confidence Comes from Experience: Experiential Confidence Estimation from Reasoning to Agents Abstract Reliable confidence estimation is increasingly central to the trustworthy deployment of language models: a calibrated estimate of the probability that an output is correct decides what to ship, what to escalate, and what to retry. Existing confidence estimators,… 16 arXiv — Machine Learning research 28d ago Beyond Static RAG: An Adaptive, Tri-Metric Routing Framework for Efficient Long-Context Inference on Commodity GPUs arXiv:2609.17564v1 Announce Type: new Abstract: Deploying retrieval-augmented generation (RAG) on commodity GPUs such as the NVIDIA T4 (16 GB VRAM) exposes a practical failure mode we call the Compression Paradox: neural prompt compression can add key-value (KV) cache contention… 7 arXiv — Machine Learning research 28d ago Bias Amplification in Multi-Agent Network: How Biased Agents Shape Opinions and Rhetoric arXiv:2609.18306v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed in applications involving interaction between agents, where their output plays a role in collective reasoning and decision-making processes. Despite significant research into… 16 arXiv — Machine Learning research 28d ago The evolution of sex for artificial intelligence: a population-genetic framework for multigenerational model populations arXiv:2609.18560v1 Announce Type: new Abstract: Some aspects of AI development resemble a population process in which models are specialised, retrained on the output of peers, or combined by averaging weights. These practices lead to generations of models, in the biological… 18 arXiv — Machine Learning research 28d ago Weakening Neurons: An Input-Output Functionality in Transformers with Outsize Influence arXiv:2609.18612v1 Announce Type: new Abstract: We analyze the learned input-output behavior of GLU-based neurons in large language models (LLMs). We propose a simple analysis method: For each neuron, we compute the cosine similarities between its input (reading) and output… 7 arXiv — Machine Learning research 28d ago WaveTLM: Reliable Time-Series Language Modeling through Task Compilation arXiv:2609.18812v1 Announce Type: new Abstract: Time-series language models provide a shared natural-language interface across temporal tasks, but plausible text does not guarantee reliable task outputs. Responses may appear reasonable while hallucinating the required object:… 37 arXiv — Machine Learning research 28d ago TwinMark: A Unified Watermark for Provable Survival Under Feature and Logit Distillation arXiv:2609.19011v1 Announce Type: new Abstract: We propose TwinMark, a watermarking scheme that reads a single SHAKE128 secret through two complementary linear functionals of model-output summaries: a covariance projector against the carrier-set covariance (cov-Feat) and a… 5 arXiv — NLP / Computation & Language research 28d ago Does Moral Reasoning Training Help or Hurt? Red-Teaming RL-Trained Ethical Agents with Persona Attacks arXiv:2609.17552v1 Announce Type: new Abstract: Moral-reward RL can make language-model agents more cooperative, but whether that alignment survives adversarial persona pressure is unknown. Such attacks are realistic: retrieved context, tool outputs, or multi-turn framing can… 10 arXiv — NLP / Computation & Language research 28d ago Confidence Comes from Experience: Experiential Confidence Estimation from Reasoning to Agents arXiv:2609.17708v1 Announce Type: new Abstract: Reliable confidence estimation is increasingly central to the trustworthy deployment of language models: a calibrated estimate of the probability that an output is correct decides what to ship, what to escalate, and what to retry.… 35 arXiv — NLP / Computation & Language research 28d ago A Calibrated Instrument for Measuring How Inference Optimizations Affect Output Quality arXiv:2609.18005v1 Announce Type: new Abstract: Large language model optimization is an active research area, spanning quantization of model weights, early-exit methods for skipping layers, and speculative decoding. Each track uses its own quality measures, typically an… 35 arXiv — NLP / Computation & Language research 28d ago Beyond Accuracy: How Procedural Traces Shift the Decision Criterion of LLM Overseers arXiv:2609.18204v1 Announce Type: new Abstract: Organizations increasingly use oversight loops where one large language model (LLM) audits another's outputs alongside procedural traces of claimed steps. A common concern about such LLM-as-a-judge pipelines is that detailed traces… 31 arXiv — NLP / Computation & Language research 28d ago SEA-LION-v4.8: A Technical Report arXiv:2609.18310v1 Announce Type: new Abstract: We introduce Nemotron-SEA-LION-v4.8, a family of Southeast Asian Languages in One Network (SEA-LION) built upon NVIDIA Nemotron 3. The family includes 30B-A3B and 120B-A12B models, with both continued-pretrained base checkpoints… 14 arXiv — NLP / Computation & Language research 28d ago Attention Dispersion as a Diagnostic Signal for Hallucination in Large Language Models arXiv:2609.18320v1 Announce Type: new Abstract: Large Language Models (LLMs) frequently exhibit hallucinations, presenting a major barrier to reliability in complex reasoning tasks. While traditional detection methods rely on output-based confidence metrics, these logits are… 35 arXiv — NLP / Computation & Language research 28d ago TalkMatrix: Generating Character Dialogue that is Both Consistent and Diverse arXiv:2609.19022v1 Announce Type: new Abstract: Candidate-based decoding typically selects a completion for each prompt independently, but many applications require a collection of outputs that satisfies global, non-decomposable requirements. We formulate this setting as… 19 arXiv — NLP / Computation & Language research 28d ago Entropy in Conversational AI: Structured Unpredictability as Inferrable Interiority arXiv:2609.19044v1 Announce Type: new Abstract: Sampling can increase response diversity without producing history-dependent behavior. We formalize a different design target, structured unpredictability, as conditional dependence between an output and a persistent hidden state… 25 r/LocalLLaMA community 28d ago Upgraded my local setup with 2 rtx pros and it's amazing. Follow up post of https://www.reddit.com/r/LocalLLaMA/s/nGMyKswrch . Thanks everyone who replied. I didn't change the specs. Might be loosing some of the memory bandwidth but will scale in future if I need to. It took me 2.5 days to build it because one of the GPU connected to… 35 r/LocalLLaMA community 28d ago I built a native Vulkan training backend for 143 modern Transformer architectures — no CUDA or PyTorch required I've been working on something that started as the training backend for my Hierarchos architecture, but it has grown into a much broader project: https://github.com/necat101/Hierarchos-Native Hierarchos Native now includes a native Rust + Vulkan backend for training and… 14 r/LocalLLaMA community 28d ago i left gpu poor range i am not GPU poor anymore https://preview.redd.it/a2tcwu9kjxph1.png?width=1173&format=png&auto=webp&s=58dfd2a26b48825d64e80cd4358584d524565bb7   submitted by   /u/paulqq [link]   [comments] 8 r/LocalLLaMA community 28d ago Enable CUDA graph for MTP draft by gaugarg-nv · Pull Request #28549 · ggml-org/llama.cpp one more MTP speedup   submitted by   /u/jacek2023 [link]   [comments] 10 llama.cpp releases dev-tools 28d ago b11007 Enable CUDA graph for MTP draft ( #28549 ) Improve CUDA graph usage for MTP Rename field Address review feedback Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/47982528 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon… 13 r/LocalLLaMA community 28d ago Qwen3.8 Flash on 12GB VRAM - 15 tokens/s Achieved steady 15 tokens/second output and 100-120 promp processing per second with 12GB GPU (RTX 5070 SFF) with Qwen3.8-Flash-Next-GSQ-RCO-GGUF at 3 bpw (IQ3_XXS which maches BF16 on AIME25). Around 76GB full gguf - while only 47GB needs to be sharded (loaded into VRAM+RAM)… 31 r/LocalLLaMA community 28d ago What's the current best LLM uncensoring method? With the recent Nvidia Huggingface acquisition and frontier AI labs screaming about safety and putting guardrails everywhere, I think it's important that we have local models that aren't affected by arbitrary guardrails set during training. To be clear, this post NOT about… 28 NVIDIA Developer Blog official-blog 28d ago Translating CUDA Tile Operations from Python to Rust Using Agentic AI cuTile Rust (cutile-rs) is a tile-based system for safe, idiomatic GPU kernel authoring in the Rust programming language. Extending the Rust ownership model to... 34 r/LocalLLaMA community 28d ago Qwen3.8-27B uncensored Q6_K at 156K context on one RTX 5090, 140-190 tok/s with DFlash2 Setup for one long agentic coding session (tools + vision) on a single 5090: Qwen3.8-27B RVN Heretic (ARA abliterated) at Q6_K, 159744 context , 1 slot, q8_0 K/V, DFlash2 speculative decoding, vision projector on the GPU, Sharp chat template. All numbers measured 2026-09-16.… 18 r/LocalLLaMA community 28d ago Qwen3.5 4B + grabbing logits is almost "Jev"? Or even just Qwen Reranker? Saw the new Introducing System One Models & Jev - TypeSafe AI Blog Jev model which just outputs probabilities given choices and I thought it sounded a something the Qwen reranker models could do? Then I tried implementing it and got even better results with Qwen 4B + just assign… 7 TechCrunch — AI news-outlet 28d ago Robots are waiting for a ChatGPT moment: Nvidia’s Les Karpas explains why at TechCrunch Disrupt 2026 The robotics industry is still waiting for their breakthrough into day-to-day life. Nvidia's Les Karpas has an answer as to why at TechCrunch Disrupt 2026. Register before September 25 to save up to $200 on your pass. 9 llama.cpp releases dev-tools 28d ago b11002 CUDA/HIP: improve access patterns in im2col ( #28013 ) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/47937267 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS… 10 r/LocalLLaMA community 28d ago Apple May Return to Server Market With Nvidia Technology Apple is considering offering an AI server built around "M8" series chips and has discussed incorporating Nvidia networking hardware, The Information reports. The system would be sold to outside customers, potentially bringing Apple back into a business it left behind when it… 30 Page 10 of 10 · 500 articles ← Newer
