r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 6d ago

Qwen 3.8 27B, just wanted to say thanks to you guys

I commented on another Qwen 3.8 27B post that I was frustrated getting anything to work. You all gave some great comments. I nuked openwebui and straightened out my llama.cpp docker config. 1 hour of work and I have a model I can chat with, connected to my HomeAssistant server,…

6
r/LocalLLaMA community 6d ago

ConvRot Quant method now in llama-cpp-turboquant

It started here , and now https://github.com/TheTom/llama-cpp-turboquant/ has it. Imagine a Q6 quant with nearly Q8 KLD/PPL. Q6_CR and Q5_CR have a slight improvement over their base counterparts. Also while you are there check out --moe-cache auto to help improve running MoE…

9
r/LocalLLaMA community 6d ago

clean your dust bunnies

Qwen 3.5 opus 4.6 distilled said I need to give him some maintenance. Featuring the wolfbox   submitted by   /u/TinFoilHat_69 [link]   [comments]

35
r/LocalLLaMA community 6d ago

Any upcoming models to be excited about?

I'm kind of new to the community and this sub is my only source of information, so I'd thought of asking if there are any upcoming models you guys are looking forward to.   submitted by   /u/zippydazoop [link]   [comments]

14
r/LocalLLaMA community 6d ago

you can now use MTP in GLM-Air

If anyone still remembers GLM-4.5-Air from last year, you can now get a nice speedup by enabling MTP in llama.cpp. It is a 106B MoE with only 12B active parameters, which makes it interesting for machines with lots of memory but limited compute, such as Strix Halo or DGX Spark.…

19
r/LocalLLaMA community 6d ago

Qwen 3.8 27b helped me with something unique that Opus 4 couldn't - Firmware + Software preservation and emulation on an early 2000's ARM based POS system

Hi all, I made a post regarding how much Qwen 3.8 has improved over 3.6: https://www.reddit.com/r/LocalLLaMA/comments/1vqm51f/long_review_qwen_38_27b_is_very_good_at_tapping/ I made a very thorough write-up of how Qwen 3.8 compared not only to 3.6, but frontier models when it…

5
r/LocalLLaMA community 6d ago

We quantized Qwen 3.8 27B and compared the quants on an RTX 6000

Me and my team made Atomic Dynamic GGUF quants for Qwen 3.8 27B, so we wanted to see the difference between them by giving each quant the same voxel island creation task First of all we were surprised at how well Qwen 3.8 27B handled the 3D scenes in general, though part of that…

15
r/LocalLLaMA community 6d ago

Nvidia Customers Notified About AI-Related Price Hikes Above 15%

  submitted by   /u/fallingdowndizzyvr [link]   [comments]

13
r/LocalLLaMA community 6d ago

New qwen3.8:27b on a 39k line C to single-file HTML / three.js port

I was just curious how the new qwen3.8:27b does on a hard C to HTML porting job against Opus 5 in a default Claude Code. The job: my fun side project is a procedural shooter in a single C file. Port it to a single-file html / three.js with one bot. One prompt, no follow-ups, no…

31
r/LocalLLaMA community 6d ago

Benchmark results: what is the best and fastest engine to run Qwen3.8-27B on macOS

WARNING: I think I made a mistake with omlx, and its results below seem to be without any kind of acceleration. I am rerunning them with Lightning MTP and ANE support for prefill. Will update the post once done. The new Qwen 3.8 27B is fantastic for local agentic use. The…

5
r/LocalLLaMA community 6d ago

Has anyone tried agent-lightning?

  submitted by   /u/MrMrsPotts [link]   [comments]

14
r/LocalLLaMA community 6d ago

Ling Tiny, King of Speed

Ling Tiny has now replaced Gemma4-12B in my rig as an auxiliary model doing hindsight operations. This is on a 4060Ti, which is a reasonable GPU available out there, and the speed is phenomenal. Don’t enable MTP, set up the vLLM fork for BailingMoE3. Hope this is useful to…

27
r/LocalLLaMA community 6d ago

1/100 → 44/100: fine-tuning a 450M VLM on 50K browser screenshots

  submitted by   /u/ButtercupLyn100 [link]   [comments]

12
r/LocalLLaMA community 6d ago

Qwen 3.8 27B for actual local programming

Most YouTube benchmarks only show trivial tasks like generating landing pages or simple Three.js games. Is a local model like Qwen 3.8 27B actually capable of real-world systems programming—such as building GTK4 or Qt 6 applications in Rust or C++ with external libraries?…

10
r/LocalLLaMA community 7d ago

I trained a game music generator

I trained a instrumental game music generator. The 1.2B DiT was trained on 1 cloud H100 from scratch in 8 days; I used the VAE from Stable Audio 3. https://huggingface.co/Localsong/Localsong https://huggingface.co/Localsong/Localsong/tree/main/samples…

23
r/LocalLLaMA community 7d ago

Qwen3.5-9B Triple-Loop

I was fascinated by Nanbeige's outstanding performance for its size, so I started digging into how much a model can improve its own representation just by looping over itself (for fun). My prototype was a Qwen3-0.6B with a full dual loop in the middle layers, inspired by the…

15
r/LocalLLaMA community 7d ago

Qwen3.8-27B NVFP4 with vision + 451K token KV-cache on one RTX 5090 (power limited to 400W) at 120 tokens/s average

Hello, So I've been trying lots of combinations in that never-ending landscape of options and settings. I wanted a proper quant of 3.8 27B running as fast as possible on my 5090 at 400W, with vision and with as much KV-cache as possible and with concurrency enabled (aiming at 3…

25
r/LocalLLaMA community 7d ago

GMKtec is going to launch new hardware with Ryzen AI Max+ PRO 495 at IFA Berlin 2026

  submitted by   /u/fairydreaming [link]   [comments]

26
r/LocalLLaMA community 7d ago

DeepSeek V4 Flash on an M2 Ultra: repacked to 141 GiB losslessly, smaller than the Q4 GGUF, at 25.8 t/s (42 t/s peak)

This is one more vibe slopped custom optimization for, in this case, my hardware (m2 ultra 60 cores, 192gb). It is just a fork from llama.cpp with a few changes, it achieves: - DeepSeek V4 Flash, no kv cache quant - 141GiB model, byte-identical lossless, smaller than the public…

13
r/LocalLLaMA community 7d ago

What is your worst sandboxing fail?

I am wondering if I am too paranoid about sandboxing the commands that come out of LLMs. It really makes my eyes twitch when I see that some IDEs, even commercial, tend to forget that they have to execute things in sandboxing and have such a brittle security model. But on the…

20
r/LocalLLaMA community 7d ago

Best model you can run on a 16gb phone?

Qwen 3.5 9B q6?   submitted by   /u/ikkiyikki [link]   [comments]

26
r/LocalLLaMA community 7d ago

Don't want to be this guy, but I need Qwen 3.8 35B A3B

Qwen 3.8 27B is great, however it takes me ages to do tasks on xhigh. I need Qwen 3.8 35B A3B. It'll be a little dumber but faster. I am also aware of the fact that 27B gets its "intelligence" from the long thinking time. I therefore assume that 35B would also be a long-thinking…

26
r/LocalLLaMA community 7d ago

Qwen3.8-27B KLDs

I have been lurking on reddit for like 6 years, and this is the first time posting, so apologies for the mistakes. I'm not a native English speaker, so sorry for the bad English too. I used LLM to translate / refine my post, including images. This is my attempt to compare the…

4
r/LocalLLaMA community 7d ago

I hosted Kimi K3 (2.8T parameters) using 8 B300s. 92 tok/s, $190 per million tokens

What I ran: 8x B300 on Modal, $56.79 per hour, vLLM, tensor parallel 8, native MXFP4 Cold boot ~27 min (1.56 TB load, JIT, 51 CUDA graph captures) TTFT 0.92 to 1.02 s, decode 92 tok/s steady, 83 tok/s average over 4 prompts $190 per million output tokens. One clean run is about…

10
r/LocalLLaMA community 7d ago

AMD Users: Have you tried the llamma.cpp AMD-Ecosystem branch? Up to 2x PP Speed

AMD has it's own llama.cpp branch: https://github.com/AMD-Ecosystem/llama.cpp And despite the Deprecation warning it's actively maintained (things are later upstreamed to the normal llama.cpp). What i noticed with my Strix Halo: It has some interesting new patches (if you use…

32
r/LocalLLaMA community 7d ago

i finally switched from windows to linux and got a 30-50% boost in speed.

This is amazing. All I did was switch from llamacpp on windows to vllm on linux.   submitted by   /u/unraveleverything [link]   [comments]

29
r/LocalLLaMA community 7d ago

DeepSeek Harness is Insanely Good

I don't know about you guys, but Deep-seek harness is insane. It's not focused on being a coder agent, it's webUI made it very easy to just checkin from time to time, and the best part? Why it's better than Hermes? It wasn't frustrating at all to setup. ZERO. NADA. Progressive…

14
r/LocalLLaMA community 7d ago

Nvidia Poolside deal to compete with Chinese Open Weights

Nvidia is investing $1 billion in Poolside and paying $6 billion to license its technology and hire most of its engineers. Over 100 Poolside staff will move to Nvidia to work on Nemotron. Good news for us!   submitted by   /u/mrgreatheart [link]   [comments]

38
r/LocalLLaMA community 7d ago

Best harness for Qwen 3.8 27b ?

I’ve tried Open Code , Codex and Qwen Code and so far Qwen Code seems to work the best for me , the only complaint I have about it is that the CLI interface is a bit rough / sloppy .   submitted by   /u/HelloWorld-Print [link]   [comments]

35
r/LocalLLaMA community 7d ago

Qwen 3.8 27B is a game changer.

Our devs got their hands on it a few days ago. One wired it into Codex to compare with GPT Luna, our usual workhorse right now for its cost effectiveness. Another tried it out on one of our OCR pipelines. It's comparable to Luna for coding and ***OCR quality appears to be better…

18
r/LocalLLaMA community 7d ago

“The All Spark” Cluster: Upgrading from 16 - 36 DGX Sparks

Earlier this year I posted about building what at the time I believe was the first 16x DGX Spark Cluster. I’m now adding 20 more Sparks to the cluster in my homelab server rack, giving me 4.6TB of unified memory. • 36x Sparks • 1x 200Gbps FS 24 x 200Gb QSFP56 + 8x 400Gb Switch •…

18
r/LocalLLaMA community 7d ago

# Qwen3.8-27B — One Week Later: The r/LocalLLaMA + r/LocalLLM Verdict

Companion to the Qwen 3.8 Release Megathread . Compiled from ~2,000 posts scanned across both subs, with deep reads of the 45 highest-signal threads (560 posts and comments), Aug 15–22, 2026, plus independent X benchmarks. Every number is attributed to the poster's stated…

16
r/LocalLLaMA community 7d ago

I fine tuned Gemma 4 12B for a 2.7x improvement on tool calling because I can't fit anything else comfortably into my 16 GBs of Vram

Gemma 12B is obviously a very well trained model, I always thought the fine tuning they did on it wasn't really cut out for agentic coding. From my own experiences it struggles to use the tools it's given from Github Copilot and is also very inept at the cli too. So I thought…

21
r/LocalLLaMA community 7d ago

Closed AI has been real quiet since Qwen 3.8 27B dropped.

This is something I've noticed. Back when GLM 5.2 and Kimi K3 launched, there was a media push on pushing how dangerous open source models are. We know why they were doing this; these open models are good enough to devalue paid closed models. I think the fear mongering was to…

16
r/LocalLLaMA community 7d ago

Has anyone actually made 64k feel like 300k+ with recursive local agents?

I'm running Qwen 3.8 27B locally on a single GPU. I can push the context to 131k, but I'd rather run it faster at 64k if the agent can manage context properly. What I have in mind is pretty simple: one model stays loaded the whole time main agent gets 64k when something is too…

32
r/LocalLLaMA community 7d ago

Tested in Coding: Q8_K_XL Qwen3.8 27B vs BF16 Qwen3.6 27B

Further to my last post, https://www.reddit.com/r/LocalLLaMA/comments/1vldngi/tested_in_coding_bf16_muse_glimmer_vs_bf16_qwen36/ , I bring this requested comparison of Qwen3.8 27B. Context: Both models are running at full FP16 KV-cache. Due to RAM limitations, BF16 Qwen3.6 was…

23
r/LocalLLaMA community 7d ago

Best harness for long autonomous tasks

I read a lot of posts like "I've given this prompt to my Qwen 3.8 27b and it one-shotted me a 3d game after 24 hours". I wonder, what's the best harness for tasks like this? It has to have auto compaction and probably a memory system, right? And excellent computer use and self…

37
r/LocalLLaMA community 7d ago

3 experiments running dsv4-flash-0731 q4+ quants on 128GB RAM + ~60 GB VRAM (with a quite bad pcie infra) with an acceptable tgs and relatively acceptable pp speed

The post describes some experiments I had while trying to desperately run deepseek-v4-flash-0731 4 bit+ quants on my machine which is supposed to support only q2 quants of the model, a or 2.xx bpw quants at best. Long story short , I wanted to have my tgs in the high twenties…

30
r/LocalLLaMA community 7d ago

Create tts voice from actual animal sound recording

In short, I want to create voices for my real chickens that I'm creating generated videos of. Ultimately I would like to create voices to be used in a tts application that are based on their real "voice patterns", as though the voice was being made with their own vocal chords. I…

26
r/LocalLLaMA community 7d ago

I benchmark DFlash 2 (PR build) in llama.cpp on Qwen 3.8 27B against all speculative methods for 3 days. 2.26x on 100 real coding prompts, 4.68x with one n-gram drafter on top. Up to 8x on specific cases.

Hey guys, Inco AI shipped DFlash 2 a few days ago with a drafter for Qwen 3.8 27B and a llama.cpp PR. I built the PR and ran it against plain decoding, MTP, the n-gram lookup drafters, and my July DFlash 1 numbers on Qwen 3.6 27B for 3 days. One RTX PRO 6000, concurrency 1,…

26
r/LocalLLaMA community 7d ago

New 100B Liquid AI model coming soon

Liquid AI currently possesses among the fastest LLM architectures around, and some of the best SLMs (in terms of utility IMO) around, so I'm very excited to see what a potential 100B LFM (3?) model would look like! Link to the poll:…

13
r/LocalLLaMA community 7d ago

GLM and I created a llama.cpp fork optimized for AMD GFX906 (Mi50, Mi60, Radeon VII, GCN HIP) - Machine Learning, LLMs, & AI

I felt the need to share this here. Looking for feedback.   submitted by   /u/milpster [link]   [comments]

31
r/LocalLLaMA community 7d ago

Single RTX 5090: Qwen3.8-27B NVFP4 at a real 262K context in vLLM — 77 tok/s short-context, 64.7 tok/s at 128K

This is the Qwen3.8-27B setup I actually use every day on one RTX 5090. I wanted to write it down with enough detail that another 5090 owner can reproduce it instead of guessing which memory knobs I used. The short version: the full 262,144-token window fits together with…

25
r/LocalLLaMA community 7d ago

Watching that wattage, in your terminal.

Released today: version 1.3 of energygraph Zero build dependencies, lightweight tool for live views of the power-consumption. Version 1.3 adds support for dGPUs from nvidia, intel, amd. Depending on vendor support, you can also get the consumption by your cpu cores, iGPU,…

4
r/LocalLLaMA community 7d ago

I forked Ninfer 3090 and converted it to run on the CMP170HX - doubled my Qwen3.6-35B from llama.cpp

Good afternoon, everyone! I wanted to show the work I've been doing around porting Ninfer over to the CMP170HX ( Github ) So, first, I do want to call out the amazing work that Neroued , Sergiuszm and specifically Don-Chad have all done, to not take away from vLLM, Llama.cpp and…

37
r/LocalLLaMA community 7d ago

Current best model for narrative, chat, prompt creation (so basically everything except agentic coding)? - 5090

Im looking to set up a new local llm (probably on unsloth studio as that seemed to be doing pretty well last time I tested it). This one won't need to do agentic coding or app building or anything (not this time) but instead more 'text' based tasks such as - being given…

10
r/LocalLLaMA community 7d ago

Fixed the MTP head on Ornith1.5 35B A3B. +3% TPS -33% wall clock

I love the Ornith 35B local models, 1.0 has been running my HAM radio rig for me. I have a hackRF receiver and a 5 watt quansheng portable the both run headless through the PC. I tried out the new Ornith1.5 build and it was faster and more accurate than 1.0. I read the threads…

30
r/LocalLLaMA community 7d ago

How to remove trendy speech from llms?

For example: Instead of saying: "I created this new ID" It says: "I minted this new ID" Instead of: "This alternative path is available" It says: "this escape hatch is available" This speech is so nonsensical and annoying. Just. Speek. Literally ... OR NORMALLY. Where did LLMs…

19
r/LocalLLaMA community 7d ago

Your own GGUF

Hello, I have a few questions that I can't seem to find a clear answer to. Does it make sense to make your own GGUF? I noticed that when I compile llamacpp (vulkan or rocm), the processing and generation is a bit better, does it work similarly with doing GGUF yourself? If I use…

11
r/LocalLLaMA community 7d ago

This is a great sub, regardless of what complaints people have about it.

This is a genuine community of real generally respectful adult human beings. Despite the enthusiasm all of you have for local AI, you can recognize that there are times when local LLMs are flawed, and even how practical they are to use for the majority of people to use. Go over…

12

Qwen 3.8 27B, just wanted to say thanks to you guys

ConvRot Quant method now in llama-cpp-turboquant

clean your dust bunnies

Any upcoming models to be excited about?

you can now use MTP in GLM-Air

Qwen 3.8 27b helped me with something unique that Opus 4 couldn't - Firmware + Software preservation and emulation on an early 2000's ARM based POS system

We quantized Qwen 3.8 27B and compared the quants on an RTX 6000

Nvidia Customers Notified About AI-Related Price Hikes Above 15%

New qwen3.8:27b on a 39k line C to single-file HTML / three.js port

Benchmark results: what is the best and fastest engine to run Qwen3.8-27B on macOS

Has anyone tried agent-lightning?

Ling Tiny, King of Speed

1/100 → 44/100: fine-tuning a 450M VLM on 50K browser screenshots

Qwen 3.8 27B for actual local programming

I trained a game music generator

Qwen3.5-9B Triple-Loop

Qwen3.8-27B NVFP4 with vision + 451K token KV-cache on one RTX 5090 (power limited to 400W) at 120 tokens/s average

GMKtec is going to launch new hardware with Ryzen AI Max+ PRO 495 at IFA Berlin 2026

DeepSeek V4 Flash on an M2 Ultra: repacked to 141 GiB losslessly, smaller than the Q4 GGUF, at 25.8 t/s (42 t/s peak)

What is your worst sandboxing fail?

Best model you can run on a 16gb phone?

Don't want to be this guy, but I need Qwen 3.8 35B A3B

Qwen3.8-27B KLDs

I hosted Kimi K3 (2.8T parameters) using 8 B300s. 92 tok/s, $190 per million tokens

AMD Users: Have you tried the llamma.cpp AMD-Ecosystem branch? Up to 2x PP Speed

i finally switched from windows to linux and got a 30-50% boost in speed.

DeepSeek Harness is Insanely Good

Nvidia Poolside deal to compete with Chinese Open Weights

Best harness for Qwen 3.8 27b ?

Qwen 3.8 27B is a game changer.

“The All Spark” Cluster: Upgrading from 16 - 36 DGX Sparks

# Qwen3.8-27B — One Week Later: The r/LocalLLaMA + r/LocalLLM Verdict

I fine tuned Gemma 4 12B for a 2.7x improvement on tool calling because I can't fit anything else comfortably into my 16 GBs of Vram

Closed AI has been real quiet since Qwen 3.8 27B dropped.

Has anyone actually made 64k feel like 300k+ with recursive local agents?

Tested in Coding: Q8_K_XL Qwen3.8 27B vs BF16 Qwen3.6 27B

Best harness for long autonomous tasks

3 experiments running dsv4-flash-0731 q4+ quants on 128GB RAM + ~60 GB VRAM (with a quite bad pcie infra) with an acceptable tgs and relatively acceptable pp speed

Create tts voice from actual animal sound recording

I benchmark DFlash 2 (PR build) in llama.cpp on Qwen 3.8 27B against all speculative methods for 3 days. 2.26x on 100 real coding prompts, 4.68x with one n-gram drafter on top. Up to 8x on specific cases.

New 100B Liquid AI model coming soon

GLM and I created a llama.cpp fork optimized for AMD GFX906 (Mi50, Mi60, Radeon VII, GCN HIP) - Machine Learning, LLMs, & AI

Single RTX 5090: Qwen3.8-27B NVFP4 at a real 262K context in vLLM — 77 tok/s short-context, 64.7 tok/s at 128K

Watching that wattage, in your terminal.

I forked Ninfer 3090 and converted it to run on the CMP170HX - doubled my Qwen3.6-35B from llama.cpp

Current best model for narrative, chat, prompt creation (so basically everything except agentic coding)? - 5090

Fixed the MTP head on Ornith1.5 35B A3B. +3% TPS -33% wall clock

How to remove trendy speech from llms?

Your own GGUF

This is a great sub, regardless of what complaints people have about it.