r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 3h ago

Qwen3.8-Flash-Next NVFP4 Day-3 support for 4xV100

RadixArk/Qwen3.8-Flash-Next-NVFP4 is now supported in SGLang-V100. 4 V100 32GB running full context. A little more than 50 GB ngram offloaded to system RAM. Prefill around 4000tks flat and decode around 60 tks all the way to the end of the 256k context. Prompt TTFT no MTP TTFT…

38
r/LocalLLaMA community 3h ago

Qwen3.8-Flash-Next optimised for Macs

Running on a M1 Max 64 GB: - SSD streaming for tensors - SSD streaming for engrams - SSD streaming for MTP How is it possible? * Custom Q4 quant: benchmarked all metal tensors then picked and spliced tensors from multiple Unsloth and AtomicChat quants to achieve best…

36
r/LocalLLaMA community 4h ago

It's official! 192GB Framework

Just noticed this on the website. At their current price tiers for the memory SKUs (32, 64, 128) I'd expect this to be ~ 4.5k for the motherboard. The PCIe slot will be open at the back as well - that's what I've heard. Maybe they make it capable of delivering 75W as well? New…

12
r/LocalLLaMA community 4h ago

an unscientific qwen 3.8 flash next and glm 5.3 flash comparison

I stole the reference image from a recent post on r/stablediffusion , and then asked both qwen 3.8 flash next (q4 K XL) and GLM flash (oQ4e MLX) to choose try to reproduce it into a "video game or tech demo" as closely as possible, iterating over a period of (up to) about an…

9
r/LocalLLaMA community 5h ago

Ran Qwen3.8-Flash-Next (79 GB, 2-bit) at 350K ctx for 3.5 hours on a 128 GB M5 Max — speed vs context depth, 100 turns, one graph

Setup: MacBook Pro M5 Max, 128 GB unified, macOS 26.5.2 · llama.cpp b10686 (Metal, 12 threads, batch 2048, flash-attn, kv-unified, ngram-mod spec decode) · Qwen3.8-Flash-Next UD-Q2_K_XL (Unsloth), 78.9 GB · 358,400-token context slot via YaRN from the native 262,144, fp16 KV.…

13
r/LocalLLaMA community 10h ago

Nemotron-3.5-Lightning at 11.77 GiB, a 16 GB option for a model that didn't have one

TL;DR: Every public low-bit GGUF of this model is secretly ~4.70 bpw. Shim the rows to 256 and it becomes a real 3.07 bpw / 11.77 GiB file that runs 262K context on 16GB. Needs patched llama.cpp — not LM Studio or Ollama. In the AtomicChat HuggingFace repo it says "There is…

4
r/LocalLLaMA community 10h ago

Humaneval benchmark for Deepseek V4 Flash 0731 vs GLM5.3 Flash on 2x DGX Spark setup

I have a 2 DGX Spark setup recently and I have been happily running Deepseek V4 Flash 0731. Since the release of GLM5.3 Flash and Qwen 3.8 Flash Next this week, a lot of folks are still waiting to see what model to run given their own hardware situations. I am very interested in…

33
r/LocalLLaMA community 14h ago

Any current Voice2Voice AI model that runs locally that’s good?

You guys remember sesame AI? With their really good AI voice model? Obviously ChatGPT has their voice model that’s also really good. Is there any smaller local variant that runs on like consumer grade gpu‘s (12,16 24gb?) I think NVidia released something but I didn’t really…

25
r/LocalLLaMA community 14h ago

Tenstorrent Qwen3.7-27b Benchmarks

I saw someone here posted about getting a Tenstorrent QuietBox 2, and I wanted to look into it the hardware. It's very difficult to find any benchmarks, but I managed to find some from an employee. The machine it was benchmarked on has 2 p300c's, their top of the line card,…

28
r/LocalLLaMA community 14h ago

67-84 t/s DeepSeek flash v4 off 2x GX10s

Finally achieved usable results with 2 gx10 at over 65 tokens a second sustained. The 2570 prompt eval is really crucial for me as well. Overall stoked 10/10 edit: I followed this setup with 2 ASUS GX10 DGX computers :)…

29
r/LocalLLaMA community 14h ago

llama.cpp Open PRs list - CPU/RAM/Disk/Hybrid Related - Better for CPU-only & Hybrid inference

Folks! We're just 50 PRs away from more faster inference . Hopefully by end of year. Experts!, please chip in there. List of Open/Ongoing PRs(and also Discussions) related to CPU/RAM/Disk/Hybrid: [Discussion] RFC: MoE expert cache, VRAM caching of hot CPU-resident experts with…

9
r/LocalLLaMA community 15h ago

(NInfer Fork) I wanted to have a 1M context Qwen-3.8 27B, tp2, dual 5090s

Hey! I forked NInfer (a from-scratch C++20/CUDA inference engine for Qwen models) and added two things: tensor-parallel across two GPUs, and YaRN ×4 rope scaling. Together they let Qwen3.8-27B NVFP4 run a 1,048,576-token context on two consumer 5090s — 27.4 GB per card, no…

21
r/LocalLLaMA community 15h ago

This finance-model benchmark card is more useful for what it discloses than for who "wins"

The official benchmark card for Ling-3.0-flash-Fin is a useful reminder that the unit being tested is rarely just “the model.” The release says most runs used temperature 1, top_p 0.95 and the highest available reasoning effort. FinFIRST and FinSearchComp Verified used a common…

22
r/LocalLLaMA community 18h ago

Qwen3.8 Flash Quants

~20–30GB smaller than Unsloth/AesSedai Q4 at similar PPL. After several days of testing I released a set of mainline-compatible imatrix quants for Qwen3.8-Flash-Next. Goal: same quality band as the popular Unsloth / AesSedai Q4 builds, less disk and RAM. Savings are roughly…

32
r/LocalLLaMA community 19h ago

Tencent compressed Hy4-preview from 1.5TB to about 200GB GGUF and kept about 98% performance.

  submitted by   /u/RedditUsr2 [link]   [comments]

16
r/LocalLLaMA community 19h ago

Someone tested various Models on the Political Compass test...

  submitted by   /u/Thrumpwart [link]   [comments]

4
r/LocalLLaMA community 19h ago

Did anyone else notice the Ornith 1.5 35B GGUFs got a "silent" update?

Hey all, I stumbled onto something funny and figured I'd share before it gets buried. I'd been running the Ornith 1.5 35B A3B GGUF (Q4_K_M) for a while, then recently a new revision showed up in my HF cache and I figured, eh, I'd pruned my old one. Fast forward a bit later when…

6
r/LocalLLaMA community 19h ago

Exo labs claiming 4.8 tb/s memory bandwidth through m5u Mac Studio clustering

Exo labs making some very exciting and interesting claims. The headline is bandwidth scales linearly on Mac Studio clusters with their solution. There is a thread over at localllm subreddit ( https://www.reddit.com/r/LocalLLM/s/qEYLOFaYwc ) where one of their employees speaks…

17
r/LocalLLaMA community 20h ago

Is it possible to run it with a combined memory setup: 16 GB VRAM + 64 GB RAM + SSD for offloading n-grams?

Hardware: rtx 5080 16 gb vram; 64 gb ram ddr5 6000hz; ssd with unlimited memory; ryzen 7 9800 x3d. OS: Windows 11 Software: I’d prefer llama.cpp, but it’s not a strict requirement; I’ll use whatever you suggest, as long as it works on Windows. My attempts to run it with…

12
r/LocalLLaMA community 20h ago

Qwen 3.8 27B at 50 tok/s with 100k Context on a 16GB GPU! (beellama.cpp)

I wanted to share my successful setup for running a Qwen 3.8 27B model with a massive context window on a consumer 16GB GPU (RTX 4070 Ti SUPER). The goal was to fit everything into VRAM without sacrificing quality or speed. 🧠 Key Components Model:…

38
r/LocalLLaMA community 21h ago

How important is it for Chinese LLMs to reach the Opus 4.8 level?

In mid-August, Ramp published spending data collected from 70,000 U.S. companies: Fable 5 ,the most powerful and expensive model in Anthropic’s lineup, accounts for just 11% of what those businesses spend on the company’s tools. The remaining 79% is worth its weight in gold.…

15
r/LocalLLaMA community 21h ago

An official 1-bit quant for Hy4??? 👀

Has anyone tried it? The results in their tweet look very promising! Sadly, I don’t have enough RAM yet… Accuracy barely moves vs BF16 📊 MCP Atlas 83.7→83.2 📊 SWE-Bench multi 82.9→81.3 📊 MRCR 81.3→81.1 📊 IFBench 73.5→72.5…

31
r/LocalLLaMA community 22h ago

Were designing a tiny autonomous research agent

This base model is only 43m parameters trained on 3m arXiv abstracts. We plan to continue pre-training and post training. If you create fine-tuning datasets or if you know of any datasets that can help shape the behavior for our goal we appreciate all contributors. The goal is…

11
r/LocalLLaMA community 22h ago

If your t/s is low enough, you can see speculative decoding with your own eyes

The other day I was trying out a distillation of DS4 Pro, and it came with MTP. It was slow as hell on my hardware, barely 2-3 t/s, BUT the speed got bumps every once in a while, and I noticed it was in moments like: United States of America First law of thermodynamics The…

19
r/LocalLLaMA community 23h ago

Qwen3.8-Next streaming - 150tps prefill, 3.6 tps decode on M5 Air

Out of curiosity, I thought I'd see if I could adapt my DSv4 streaming stack from a few weeks ago to take Qwen3.8-Next. It worked, better than I thought - it actually runs faster on my 32GB M5 than the dense 27b does (admittedly not apples to apples as I decided to use a 3bit of…

22
r/LocalLLaMA community 1d ago

Different Qwen thinking levels

  submitted by   /u/Tall_Abrocoma_3533 [link]   [comments]

27
r/LocalLLaMA community 1d ago

I always wonder how much more speed and/or context they'd be getting..

Nothing personal. I just have too much time on my hands. Probably because I spend none of it inspecting the code my agent writes, just the finished product.   submitted by   /u/_-_David [link]   [comments]

18
r/LocalLLaMA community 1d ago

Terminal Bench 4.0 just dropped, GLM-5.3 is at the same level as Fable 5, accounting for margin of error

Announcement: https://www.tbench.ai/news/terminal-bench-4-0 Leaderboard: https://www.tbench.ai/ Imo the best aspect in their announcement is their focus on rapidly iterating on TerminalBench to keep the pace up with new model releases to fight benchmark saturation. On a similar…

20
r/LocalLLaMA community 1d ago

~ 2x Speed Boost for Qwen3.8 27B on Apple Silicon

https://x.com/koc_z3/status/2093581036756025744?s=46 ~ 2x speed boost for Qwen3.8 27B on Apple Silicon ~ 1.5x speed boost for Qwen3.6 35B AЗB Tested on an M1 Max 64GB Mac using MTPLX with 262K (MAX) Context length. Qwen3.8-27B (Q4): - Decode ~ 21 TPS - Prefill ~ 83 TPS (Peak 111…

6
r/LocalLLaMA community 1d ago

Qwen3.8-27B vs Qwen3.8-Flash-Next smaller quant?

If you only had 128gb ram which one would be more "intelligent", Qwen3.8-27B (or even 3.6) or a smaller quant of Qwen3.8-Flash-Next (Q4/Q5) ? Mostly for discussions, but also interested in coding. Thanks edit: I have a 128gb Halo. By "intelligent" I mean more intelligent answers…

35
r/LocalLLaMA community 1d ago

Qwen3.8-Flash-Next + MTP on Strix Halo: Vulkan Runtime Notes

Below are the benchmark results for running Qwen3.8-Flash-Next on Strix Halo using the Vulkan backend of llama.cpp, combined with MTP model. Hardware Item Details CPU AMD Ryzen AI MAX+ 395 (16C/32T) GPU Radeon 8060S (integrated, RADV STRIX_HALO) RAM 128GB unified memory Software…

27
r/LocalLLaMA community 1d ago

Linux Ubuntu 26.04 LTS & 26.04.1 upgrade - How much Improvements?

People who moved from older version to this new one, how much improvements do you see on inference? Ex: llama.cpp performance? This new version comes with Linux Kernel 7.0. Hope AMD cards gonna enjoy additional improvements with their recent ROCm 10.0 version release.  …

14
r/LocalLLaMA community 1d ago

Saved my fiances phone with qwen 3.8 27b

this model is really something incredible, saved us like 600 dollars. my fiance is always breaking her electronics and then getting me to fix them. The other day she brought me her phone and it was stuck in a boot loop that wouldn't post. tried normal stuff, managed to get it to…

26
r/LocalLLaMA community 1d ago

50% tg increase with offloading "hot" experts to VRAM

I got a 50% performance boost (20 t/s -> 30 t/s) in llama.cpp for MoE models that don’t fit entirely in VRAM—in my case, Qwen 3.8 Flash Next. The idea is simple: instead of offloading entire layers to the GPU, I offload only the “hot” experts. I found that certain groups of…

34
r/LocalLLaMA community 1d ago

use llms to auto annotation your dataset locally

hi i make tool for this called llmog it's purpose to make llms free to - auto annotation datasets - reclassification existing yolo datasets running totally local using llama cpp or vllm or use external api you'd rather click than code. 🔗 GitHub: mohamed-em2m/llmog: framework…

36
r/LocalLLaMA community 1d ago

Tenstorrent Quietbox 2 Arrived!

Super excited to start dev on this. I've already deployed a number of galaxy systems and want to really figure out the ideal models on this architecture that I haven't discovered. It's actually quite a system for the price. 256G system mem, 128G of interconnected GDDR over the…

34
r/LocalLLaMA community 1d ago

Qwen 3.8 Flash Next ngram look up table offloaded to SSD and streamed in SGLang

Have anyone tried this yet? Looks promising, seems too good to be true with no performance loss.   submitted by   /u/Easy_Werewolf7903 [link]   [comments]

5
r/LocalLLaMA community 1d ago

AtomicChat/Qwen3.8-Flash-Next-GGUF is Really Good

specs hardware: M4 Max 128GB Studio inference engine: llama.cpp (qwen4exp branch) judge: claude-opus-4-6 AtomicChat/Qwen3.8-Flash-Next-GGUF Qwen3.8-Flash-Next is a great model I benched in my previous post , but it is very tight, since all n-grams / PLE are loaded along with the…

17
r/LocalLLaMA community 1d ago

Hot or not?

Does anyone else add active cooling to their DGX stack? Found mine was getting quite hot under extended load. This helps immensely with that so far. I will be adding some stats as they relate to comphy and DeepSeek flash this weekend. I did not create the original designs but…

27
r/LocalLLaMA community 1d ago

Is it worth running Qwen 3.8 Flash Next on 4x3090 vs 27B?

Can someone please tell me if it's worth running Qwen 3.8 Flash Next on 4x3090 yet over 27B? 27B is good but damn it is indecisive. I am getting frustrated watching it get "so close" to solving a problem, only to do another 2 hours of "let me just check/prove/etc" It looks like…

30
r/LocalLLaMA community 1d ago

Today I hit 181 toks/s (aggregate) on Qwen3.8-Flash-Next on 2x DGX Sparks

Hey all, and hello fellow DGX Spark-ers! Today I managed some pretty crazy numbers: 181 tok/s aggregate on 2× DGX Spark on Qwen3.8-Flash-Next at 512Kcontext (2.8M kvc) I hit 181 tok/s aggregate today across a multi-agent fleet on a 2-node DGX Spark cluster. Single-stream decode…

22
r/LocalLLaMA community 1d ago

[Release] SOTA GGUFs for Qwen3.8-27B: GSQ-RCO at 2.5 to 3.0 bpw

We're releasing Qwen3.8-27B quantized with our newest methods, GSQ + RCO. Higher-quality models, same file size, now with the search and the quantizer both learned. What's inside: GSQ (Gumbel-Softmax Quantization): post-training scalar quantization that jointly learns grid…

32
r/LocalLLaMA community 1d ago

I audited 443 GGUF quants across 25 repos. 64 of them can't be the quant their filename claims.

TL;DR: k-quants need tensor rows divisible by 256. When they aren't, llama-quantize quietly swaps in a ~4.5 bpw type and the file keeps its low-bit name. I audited 443 quants across 25 repos; 64 are affected. On Nemotron-3.5-Lightning all four IQ2 rungs are the same 4.58 bpw…

28
r/LocalLLaMA community 1d ago

GLM-5.3 on HF Viewer

GLM-5.3 is now ready to visualize in HF Viewer! Note again that the architecture is unchanged since GLM-5.2! Crazy improvements just from improved training! The graph visualization becomes quite complex at the expanded levels, with sparse attention, MoE routing, shared indexers,…

35
r/LocalLLaMA community 1d ago

Breeze-TTS-2 initial impressions: genuinely 'frontier' TTS

You can test it out on breezblue's playground or use it locally, its only ~7GB.   submitted by   /u/Gohab2001 [link]   [comments]

20
r/LocalLLaMA community 1d ago

I benchmarked 9 open models on spotting fake sources during agentic search (DeepSeek V4, Qwen 3.8, Nemotron 3 Ultra)

I built a benchmark called EchoNet. An agent gets a factual question, then searches a syntheic web I made before answering. Some of that web is seeded with misinformation: one fake page, a fake page ranked first in search results, the same fake claim copied across many pages, a…

27
r/LocalLLaMA community 1d ago

ROCm 10.0: A Decade of Open Compute, Built for the Age of Agentic AI

Their last version 7.14 was released just a month ago. llama.cpp PR(waiting for approval) for Version 10.0 https://github.com/ggml-org/llama.cpp/pull/27803 Hope this version comes with more boost & improvements.   submitted by   /u/pmttyji [link]   [comments]

35
r/LocalLLaMA community 1d ago

Local agentic coding Benchmark : Qwen3.8-Flash-Next NVFP4 vs 27B (and the others...)

Using https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4 and https://old.reddit.com/r/BlackwellPerformance/comments/1w04xb7/qwen38_flashnext_on_1x_rtx_pro_6000_171_ts_c1_428/ As usual, all the details in…

34
r/LocalLLaMA community 1d ago

ds4 branch with GLM 5.3 Flash support

As a happy user of ds4, I'm very excited about this branch. Ran some prompts and it seems to be working well on my M4 Max 128gb! https://x.com/antirez/status/2093349448445243873   submitted by   /u/lakySK [link]   [comments]

32
r/LocalLLaMA community 1d ago

Qwen3.8-Flash on RTX3090 + 64GB RAM (but you only need 12GB VRAM)

I've got Qwen3.8-Flash-next running on RTX 3090, Ryzen 9 3950X, a PCIe 3.0 motherboard, and 64GB DDR RAM from 2020. IQ4_XS weights, full kvarn5 context, vision on GPU, experts in host RAM, n-grams on disk. MTP works but actually slows decode down even with 80% draft acceptance,…

18

Qwen3.8-Flash-Next NVFP4 Day-3 support for 4xV100

Qwen3.8-Flash-Next optimised for Macs

It's official! 192GB Framework

an unscientific qwen 3.8 flash next and glm 5.3 flash comparison

Ran Qwen3.8-Flash-Next (79 GB, 2-bit) at 350K ctx for 3.5 hours on a 128 GB M5 Max — speed vs context depth, 100 turns, one graph

Nemotron-3.5-Lightning at 11.77 GiB, a 16 GB option for a model that didn't have one

Humaneval benchmark for Deepseek V4 Flash 0731 vs GLM5.3 Flash on 2x DGX Spark setup

Any current Voice2Voice AI model that runs locally that’s good?

Tenstorrent Qwen3.7-27b Benchmarks

67-84 t/s DeepSeek flash v4 off 2x GX10s

llama.cpp Open PRs list - CPU/RAM/Disk/Hybrid Related - Better for CPU-only & Hybrid inference

(NInfer Fork) I wanted to have a 1M context Qwen-3.8 27B, tp2, dual 5090s

This finance-model benchmark card is more useful for what it discloses than for who "wins"

Qwen3.8 Flash Quants

Tencent compressed Hy4-preview from 1.5TB to about 200GB GGUF and kept about 98% performance.

Someone tested various Models on the Political Compass test...

Did anyone else notice the Ornith 1.5 35B GGUFs got a "silent" update?

Exo labs claiming 4.8 tb/s memory bandwidth through m5u Mac Studio clustering

Is it possible to run it with a combined memory setup: 16 GB VRAM + 64 GB RAM + SSD for offloading n-grams?

Qwen 3.8 27B at 50 tok/s with 100k Context on a 16GB GPU! (beellama.cpp)

How important is it for Chinese LLMs to reach the Opus 4.8 level?

An official 1-bit quant for Hy4??? 👀

Were designing a tiny autonomous research agent

If your t/s is low enough, you can see speculative decoding with your own eyes

Qwen3.8-Next streaming - 150tps prefill, 3.6 tps decode on M5 Air

Different Qwen thinking levels

I always wonder how much more speed and/or context they'd be getting..

Terminal Bench 4.0 just dropped, GLM-5.3 is at the same level as Fable 5, accounting for margin of error

~ 2x Speed Boost for Qwen3.8 27B on Apple Silicon

Qwen3.8-27B vs Qwen3.8-Flash-Next smaller quant?

Qwen3.8-Flash-Next + MTP on Strix Halo: Vulkan Runtime Notes

Linux Ubuntu 26.04 LTS & 26.04.1 upgrade - How much Improvements?

Saved my fiances phone with qwen 3.8 27b

50% tg increase with offloading "hot" experts to VRAM

use llms to auto annotation your dataset locally

Tenstorrent Quietbox 2 Arrived!

Qwen 3.8 Flash Next ngram look up table offloaded to SSD and streamed in SGLang

AtomicChat/Qwen3.8-Flash-Next-GGUF is Really Good

Hot or not?

Is it worth running Qwen 3.8 Flash Next on 4x3090 vs 27B?

Today I hit 181 toks/s (aggregate) on Qwen3.8-Flash-Next on 2x DGX Sparks

[Release] SOTA GGUFs for Qwen3.8-27B: GSQ-RCO at 2.5 to 3.0 bpw

I audited 443 GGUF quants across 25 repos. 64 of them can't be the quant their filename claims.

GLM-5.3 on HF Viewer

Breeze-TTS-2 initial impressions: genuinely 'frontier' TTS

I benchmarked 9 open models on spotting fake sources during agentic search (DeepSeek V4, Qwen 3.8, Nemotron 3 Ultra)

ROCm 10.0: A Decade of Open Compute, Built for the Age of Agentic AI

Local agentic coding Benchmark : Qwen3.8-Flash-Next NVFP4 vs 27B (and the others...)

ds4 branch with GLM 5.3 Flash support

Qwen3.8-Flash on RTX3090 + 64GB RAM (but you only need 12GB VRAM)