r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 8d ago

Qwen 3.5 4B IQ2_XS: +16.67% Reasoning Performance From Tensor-Level Allocation

I was finally able to replicate tensor level allocation outside the Gemma family. https://huggingface.co/ByteOtter/Qwen3.5-4B-CADA-IQ2_XS After the Gemma 4 12b, e4b and gemma 3 4b results, I attempted to expand into qwen and ran into a few walls. After 2 version updates and a…

4
r/LocalLLaMA community 8d ago

web_search tool in deepseek harness needs api key from deepseek and deepseek charges you as deepseek-v4-flash usage.

I was experimenting with deepseek harness when found that even if you don't use deepseek models, you can configure the web_search tool with their api key and every hit will cost you as if you called deepseek-v4-flash model. It's a bummer. on top of that there is currently no…

38
r/LocalLLaMA community 8d ago

Anyone else tried out KV cache blending?

Idea is simple-ish in abstract: instead of running normal prefill over all of a given prompt, split it into parts - generates caches for part A and part B in isolation, concatenate the result, feed it into decode like normal. I honestly thought it'd totally fail. But I've been…

4
r/LocalLLaMA community 8d ago

Think you're going to get cheap DDR5 RAM? Think again, even if prices fall, scalper bots now outnumber shoppers 10 to 1 and will keep prices high

  submitted by   /u/DeltaSqueezer [link]   [comments]

6
r/LocalLLaMA community 8d ago

Artificial Analysis "Intelligence": A meaningless benchmark

https://preview.redd.it/84zi5nsdawkh1.png?width=2368&format=png&auto=webp&s=1109e69db807b153064b1f5b61d22cf1e9fbca05 Another user posted the benchmarks for Qwen 3.8 27B today, and while I think Qwen 27B is a really powerful model, I can't help but notice just how meaningless…

34
r/LocalLLaMA community 8d ago

Need support for llama.cpp with multi GPU

Using llama.cpp I seem to be unable to get my to GPUs working tougether correclty, so I need help somehow. Setup: 96GB RAM, one Blackwell 5000 (48GB) and one 3090 (24GB). I am trying to run the UD-Q3_K_XL quant of Deepseek4 flash which has about 120GB size. Using just the…

20
r/LocalLLaMA community 8d ago

AntLing released a dspark draft model for Ling-3.0-flash

No GGUFs yet on Huggingface though.   submitted by   /u/Ihtien [link]   [comments]

8
r/LocalLLaMA community 8d ago

Freetokens project is impressive

A new project was released yesterday and I have the opportunity to test it today. Papper: https://arxiv.org/abs/2608.16157 Github: https://github.com/FlashML-org/FreeToken My initial tests with the following setup: RTX 5080 (16 GB) DDR6 64GB AMD Ryzen 9 9950X3D I got 100tok/s on…

14
r/LocalLLaMA community 8d ago

Llama.cpp version 0.2.0 is out!

You can find the changelog and source code here: https://github.com/ggml-org/llama.cpp/releases/tag/v0.2.0 Associated pre-build is here: https://github.com/ggml-org/llama.cpp/releases/tag/b10566   submitted by   /u/PhilippeEiffel [link]   [comments]

32
r/LocalLLaMA community 8d ago

Bro wtf, Qwen Lab cooked with Qwen 3.8 27B, it's so fucking good

Context: Earlier, open-source large models like Mimo V2.5 Pro, DeepSeek V4 Pro (first version), and Kimi K2.5 used to struggle with this prompt, and Qwen 3.6 27B couldn't even render the globe properly. But now, Qwen 3.8 27B is so much better than the previous model. I hope qwen…

29
r/LocalLLaMA community 8d ago

Sharp template to NInfer: -42% output tokens, same speed

Sharp is u/peculiar-ragdoll 's system prompt that makes Qwen answer way more tersely without losing correctness. It's built on top of froggeric's fixed chat templates for Qwen; several fixes now in the v22.x templates (error-escalation tiers, false retry-loop kills, multi-system…

20
r/LocalLLaMA community 8d ago

This is why I run locally.

It was only a matter of time...   submitted by   /u/Retumbo77 [link]   [comments]

24
r/LocalLLaMA community 8d ago

16 GB VRAM purgatory discussion thread

What models and configs are we using? Please share here On windows, I am using this copium pared down model https://huggingface.co/Bucoid/Qwen3.8-27B-Uncensored-IQ4-XS-MTP-16GB-VRAM-GGUF with MTP disabled, q4 k/q4 v mmproj banished to CPU/RAM and a small ub to save whatever…

18
r/LocalLLaMA community 8d ago

How to give a local LLM/agent access to a "real" web browser

I can't seem to find a good answer to this, my Hermes agent has access to Firecrawl and some other web scrapers for content extraction, but anyone know of a way to let a local LLM drive a "real" web browser? My wife asked me to have Hermes go and look at her LinkedIn profile,…

21
r/LocalLLaMA community 8d ago

Qwen 3.8 27b - PI AGENT vs OPENCODE - another smaple

That is the second comparison and the last one. I will not be spamming again ;) Continuation from: https://www.reddit.com/r/LocalLLaMA/comments/1vu0u2v/qwen_38_27b_pi_agent_vs_opencode/ That is one of my many tests I make comparing output quality. What is more interesting using…

9
r/LocalLLaMA community 8d ago

I tried to do agenic coding with Qwen 3.8 27B 3bit quant on a macbook air m2 24gb. It took 63 hours, but amazingly, the flight simulator worked.

I used LM Studio Bionic with Qwen 3.8 27B Q3_K_S with 57k context. It took a staggering 63 hours to finish coding. After the first prompt "Create a beautiful, relaxing flight simulator in a single HTML page" taking 47.8 hours, it created an html file that showed the title screen…

15
r/LocalLLaMA community 8d ago

I'm really hoping we're in 2026's 2-month-gap between QwQ and Qwen3 right now

QwQ was genuine next-gen performance usable on local hardware, but the massive required context (it's reasoning style was akin to "if I say every possible word, I'll notice the right one!" ) kinda made it unusable for agentic coding. It was ~2 months later that Qwen3-32B came…

20
r/LocalLLaMA community 8d ago

Qwen3.8-27B different thinking levels

Even the low preset is better than Qwen 3.7 plus or Qwen3.6-27B reasoning   submitted by   /u/Tall_Abrocoma_3533 [link]   [comments]

24
r/LocalLLaMA community 8d ago

Qwen 3.8 Low and Medium are goated

Artificial Analysis just benchmarked them and the scores are crazy good, proving the earlier success wasn't only enabled by overthinking.   submitted by   /u/Eyelbee [link]   [comments]

37
r/LocalLLaMA community 8d ago

I feel like I finally graduated.

I finally made the move from LM Studio to vLLM thanks to this post https://www.reddit.com/r/LocalLLaMA/s/NmS9CgHvqz . I may not know what it all means yet but I’m going to start diving into the docs to learn as much as I can. I’m running an endpoint on each of my 3090s one for…

4
r/LocalLLaMA community 8d ago

Strix Halo (8060S / gfx1151), Qwen-3.8-27B @ Q8 and Q6 UD v3, up to 256K ctx, llama.cpp, DFlash2, vision, real workloads quality and steady performances, optimized recipes, ...

Hi fellows fully-local halos, after manually following existing guides, I decided to build an LLM API endpoint installation and optimization guide that works even when autonomously followed by my pi agent, so I can install/experiment/reinstall easily and without babysitting. Q8…

38
r/LocalLLaMA community 8d ago

Qwen3.8-27B on an RTX 5060 Ti 16GB: IQ4 vs Q8, 64K context, MTP, vision, and agent benchmarks

I’ve been testing Qwen3.8-27B as a possible replacement for the Qwen3.5-9B that I have been running on RTX 5060Ti 16G. The goal was not just maximum tokens/sec, but useful context capacity, reliable tool calling, multi-turn behavior, and vision on a single 16GB GPU for true…

34
r/LocalLLaMA community 8d ago

Qwen3.8-27B Q6 is a beast at agentic coding

A quick feedback after a really major test: nearly 20 hours of non-stop goal-oriented work with Qwen3.8-27B Q6, running across an RTX 3090 and an RTX 3060. It maintained a speed of around 60–63 tokens/s throughout the session.   submitted by   /u/Ok_Ninja7526 [link]…

30
r/LocalLLaMA community 8d ago

model: add dots3-note by ngxson · Pull Request #27060 · ggml-org/llama.cpp

dots3-note preview is the first open-weight model in the dots3 family. It is a Mixture-of-Experts model with 280B total parameters, 16B activated parameters, and support for a context length of up to 512K tokens. The model can understand text, images, video, and audio, and…

14
r/LocalLLaMA community 8d ago

Llama.cpp DSpark PC Tree Fork (up to 3%-29.5% faster!)

Hello gang, I made an implementation of DSpark PC Tree (Parent conditioned drafting tree). This is an implementation of this research paper: https://arxiv.org/abs/2608.02123 Unaffiliated, just found it and implemented it. And I have to preface: This is just a first shot, I have…

35
r/LocalLLaMA community 8d ago

Qwen 3.8 vs 3.6 27b low reasoning loops way less now

Have seen some people say Qwen 3.8 still overthinks even when reasoning is set to low. Which on my case has been way better compared to 3.6, eveb on a 3 bit quant. I think it's worth mentioning that the default is actually xhigh, so first make sure to specify it if not already.…

36
r/LocalLLaMA community 8d ago

What’s the best local AI harness for coding + general use?

So what’s actually the best local AI harness rn? I’ve read a TON about this already and somehow ended up more confused than when I started so I figured screw it, let the community decide. Right now I mainly run Qwen 3.6 35B-A3B and Qwen 3.8 27B, with Ornith 1.5 9B sometimes for…

16
r/LocalLLaMA community 8d ago

FireRedAudio & FireRedTTS3 by FireRedTeam - Huggingface

FireRedAudio: A General-Purpose Audio Language Model with Decoupled Continuous Representations for Understanding and Generation HuggingFace : https://huggingface.co/FireRedTeam/FireRedAudio GitHub : https://github.com/FireRedTeam/FireRedAudio Demo :…

33
r/LocalLLaMA community 8d ago

Ultrafast Qwen3-TTS at 34 ms Time-to-First-Audio, Handling 10 Requests Per Second [OSS]

Hey locallama! We recently open sourced a Qwen3-TTS 1.7B implementation that achieves 10 requests per second (RPS) and sub-50 ms p95 time-to-first-audio (TTFA) while maintaining real-time playback on 1 x H100. This extends to 20 RPS at sub-100 ms p95 TTFA. By adjusting settings,…

31
r/LocalLLaMA community 8d ago

I benchmarked Ox Alpha on SWE-bench Verified Mini (50 tasks): 96% resolved. Now I’m skeptical of myself.

TL;DR: The result looks TOO GOOD TO BE TRUE — that's exactly why I'm posting it. I ran Ox alpha on the entire SWE-bench Verified-Mini set with the official mini-swe-agent scaffold — the same agent used for the swebench.com Bash-Only leaderboard. Result: 48/50 = 96% resolved,…

11
r/LocalLLaMA community 9d ago

NVIDIA AVO got 100% on ARC-AGI-3. It completed all 183 levels across all 25 public environments, figuring out what to do with no instructions, explicit rules, or stated goals.

  submitted by   /u/theologi [link]   [comments]

35
r/LocalLLaMA community 9d ago

DeepSeek Harness v0.1.1 released

https://github.com/deepseek-ai/deepseek-harness/releases/tag/dsh-v0.1.1-rc.1 The DeepSeek adapter adds the multimodal visual understanding model DeepSeek-V4-Flash-Vision-Exp. It also supports configuring native image requests. Commands such as /goal and /plan can accept text and…

22
r/LocalLLaMA community 9d ago

Qwen 3.8 27b is strong even at Q3_xxs

So usually I avoid Q3 quants because I have had bad experiences with it, models were usually too degraded, so the smallest I normally do is Q4, since I only have rtx 4060 ti 16gb. But since there hasn't been a 35b-3ab released yet, I had to try it. I don't use LLMs in agentic…

13
r/LocalLLaMA community 9d ago

Buying a V100/older NVIDIA GPU? Run this to check for older memory issues

I recently bought a 32GB V100 off eBay and thought I was all set with rudimentary tests showing zero issues. But then I started seeing weird VRAM-related errors in llama.cpp. I sic'ed Claude on it, to find that ECC was disabled and that can hide small RAM issues. I believe the…

19
r/LocalLLaMA community 9d ago

When 9 of top 12 models are just exactly one.

  submitted by   /u/Beamsters [link]   [comments]

7
r/LocalLLaMA community 9d ago

Ornith-1.5-35B-A3B-NInfer - 250 tok/s, 5-8k prefill, 5090

I tried this model yesterday, and it felt to me like the best one I've tried for a local model for interactive use; the responses and reasoning are very fast, and it actually performs agentic tasks well. The speed is phenomenal. I am running this on Ninfer for Windows -…

37
r/LocalLLaMA community 9d ago

I got fed up with locked-down autocomplete, so I forked Continue and stripped it down to just tab-completion. Any model, no subscription and no remote telemetry

I kept running into the same wall. Every agentic coding plugin ships an autocomplete, and almost all of them lock it down — no model choice, or a subscription, or both. I don't want a chat panel, I don't want an agent rewriting my repo. I want ghost text that finishes the line…

8
r/LocalLLaMA community 9d ago

DeepSeek-V4-Flash-Vision-Exp

  submitted by   /u/Xhehab_ [link]   [comments]

8
r/LocalLLaMA community 9d ago

Fastest NVFP4 quant of Qwen3.8 27B out there

Here's a brand new Blackwell-native, prefill-optimized 4-bit quant that runs 50% faster on compatible hardware than a Q4 quant of the same memory footprint. And it runs 4-7% faster than other NVFP4 quants as benchmarked on RTX 5090 32GB. Quant Benchmark Speed NVFP4 pp2048 6250…

27
r/LocalLLaMA community 9d ago

Does PCIe matter much for inference?

Speaking of -sm tensor. I have 2x5060ti and i get some 40-50tps form 3.8 27B q6. I used HWinfo to see how saturated the PCIes are during inference are and as expected both the PCIe 5x16 slot and PCIe 4x4 were fully saturated. I cant help but feel like my 2nd gpu slot is a…

7
r/LocalLLaMA community 9d ago

New stealth model on OpenRouter: Ox Alph

Any idea which lab this is from? people are guessing this is a Chinese model.   submitted by   /u/Neosinic [link]   [comments]

36
r/LocalLLaMA community 9d ago

Useful > Fast

I run 2 x 3090 on a ryzen 7 with 32gb ddr4 6000. I see a lot of posts about maxing speed / context pool. I’ve done this myself with 3.8 27b. What I’m interested in though is once the dust settles and we look at utility, what balance people are striking in terms of similar setups…

5
r/LocalLLaMA community 9d ago

Qwen3.8-27B at 262K context on a Strix Halo + RTX 3090 Ti: 9.5 -> 153 tok/s, and it beats a dual-3090 vLLM box on HumanEval

Spent a while treating layer placement, KV format and llama.cpp itself as experimental variables. 159 logged experiments. Numbers first, caveats after. Hardware: AMD Ryzen AI MAX+ 395 (Strix Halo, 128 GB unified) + RTX 3090 Ti on an eGPU link. One llama.cpp process, AMD on…

37
r/LocalLLaMA community 9d ago

Fastest qwen 3.8 27b for AMD gpu?

Hey, just wondering if there are forks or exact gguf versions that give fastest prompt processing and token gen speeds for AMD gpu? Looking to run q8 or q6 Vram 96gb W7900 + w7800 both 48gb With bandwidth mismatch, tensor paralleling amd equivalent not working   submitted by…

21
r/LocalLLaMA community 9d ago

Ox Alpha stealth model: GLM5 Air, Mimo V3 or ?

To anyone who needs AIR…   submitted by   /u/Miserable-Dare5090 [link]   [comments]

27
r/LocalLLaMA community 9d ago

New stealth model Ox Alpha look like glm or mimo new model

https://preview.redd.it/stgzmbpgnnkh1.jpg?width=2179&format=pjpg&auto=webp&s=19a4fd0e1786cbcac0967e1bf1ec4898447e989a https://preview.redd.it/paub9gylnnkh1.png?width=680&format=png&auto=webp&s=c13efb53356b1b7e7826aece96914cad9252240c Ox Alpha is a frontier model built for…

30
r/LocalLLaMA community 9d ago

SenseNova U1.5-Lite full release: expert training, OPD distillation, one model at inference

Benchmarks: Benchmark U1 Preview Full Qwen-Image-Bench 47.14 55.20 (PE) 60.18 (PE) ImgEdit 3.9 4.37 4.59 GEdit-Bench-EN 7.47 8.14 8.26 Instead of just scaling up, they train task-specialized expert models for text rendering and infographics, aesthetic quality, and image editing.…

15
r/LocalLLaMA community 9d ago

Is there any interest in a self hosted open source version of manus/perplexity?

I have been working on a project for about 3 months because I found Vane to be awful and I wanted something like Manus for myself. I have a pretty solid app that's probably about ready to release, but I'm not sure if there is any desire there. It does research at different…

9
r/LocalLLaMA community 9d ago

Make Jensen Huang Sound Like Anyone. New Streaming Voice Conversion Model MeanVC2 Released!

Finally see a new voice conversion model. MeanVC2 supports cross-gender and cross-language voice conversion. 3x realtime on CPU with audio.cpp. Disclaimer: The converted voice quality of MeanVC2 is decent; the noise comes from my rough demo engineering, not the model itself.…

38
r/LocalLLaMA community 9d ago

I did it! I'm free! It's been 7 hours since I used claudecode

My Pro subscription expired today, they killed my access at 1pm local time. I'm now using Qwen3.8-27b w/ 5090m 24gb vram and pi to do everything i was doing in claudecode. The only downside is claudecode let me code without using my gpu, meaning I have to plan things now. Last…

15

Qwen 3.5 4B IQ2_XS: +16.67% Reasoning Performance From Tensor-Level Allocation

web_search tool in deepseek harness needs api key from deepseek and deepseek charges you as deepseek-v4-flash usage.

Anyone else tried out KV cache blending?

Think you're going to get cheap DDR5 RAM? Think again, even if prices fall, scalper bots now outnumber shoppers 10 to 1 and will keep prices high

Artificial Analysis "Intelligence": A meaningless benchmark

Need support for llama.cpp with multi GPU

AntLing released a dspark draft model for Ling-3.0-flash

Freetokens project is impressive

Llama.cpp version 0.2.0 is out!

Bro wtf, Qwen Lab cooked with Qwen 3.8 27B, it's so fucking good

Sharp template to NInfer: -42% output tokens, same speed

This is why I run locally.

16 GB VRAM purgatory discussion thread

How to give a local LLM/agent access to a "real" web browser

Qwen 3.8 27b - PI AGENT vs OPENCODE - another smaple

I tried to do agenic coding with Qwen 3.8 27B 3bit quant on a macbook air m2 24gb. It took 63 hours, but amazingly, the flight simulator worked.

I'm really hoping we're in 2026's 2-month-gap between QwQ and Qwen3 right now

Qwen3.8-27B different thinking levels

Qwen 3.8 Low and Medium are goated

I feel like I finally graduated.

Strix Halo (8060S / gfx1151), Qwen-3.8-27B @ Q8 and Q6 UD v3, up to 256K ctx, llama.cpp, DFlash2, vision, real workloads quality and steady performances, optimized recipes, ...

Qwen3.8-27B on an RTX 5060 Ti 16GB: IQ4 vs Q8, 64K context, MTP, vision, and agent benchmarks

Qwen3.8-27B Q6 is a beast at agentic coding

model: add dots3-note by ngxson · Pull Request #27060 · ggml-org/llama.cpp

Llama.cpp DSpark PC Tree Fork (up to 3%-29.5% faster!)

Qwen 3.8 vs 3.6 27b low reasoning loops way less now

What’s the best local AI harness for coding + general use?

FireRedAudio & FireRedTTS3 by FireRedTeam - Huggingface

Ultrafast Qwen3-TTS at 34 ms Time-to-First-Audio, Handling 10 Requests Per Second [OSS]

I benchmarked Ox Alpha on SWE-bench Verified Mini (50 tasks): 96% resolved. Now I’m skeptical of myself.

NVIDIA AVO got 100% on ARC-AGI-3. It completed all 183 levels across all 25 public environments, figuring out what to do with no instructions, explicit rules, or stated goals.

DeepSeek Harness v0.1.1 released

Qwen 3.8 27b is strong even at Q3_xxs

Buying a V100/older NVIDIA GPU? Run this to check for older memory issues

When 9 of top 12 models are just exactly one.

Ornith-1.5-35B-A3B-NInfer - 250 tok/s, 5-8k prefill, 5090

I got fed up with locked-down autocomplete, so I forked Continue and stripped it down to just tab-completion. Any model, no subscription and no remote telemetry

DeepSeek-V4-Flash-Vision-Exp

Fastest NVFP4 quant of Qwen3.8 27B out there

Does PCIe matter much for inference?

New stealth model on OpenRouter: Ox Alph

Useful > Fast

Qwen3.8-27B at 262K context on a Strix Halo + RTX 3090 Ti: 9.5 -> 153 tok/s, and it beats a dual-3090 vLLM box on HumanEval

Fastest qwen 3.8 27b for AMD gpu?

Ox Alpha stealth model: GLM5 Air, Mimo V3 or ?

New stealth model Ox Alpha look like glm or mimo new model

SenseNova U1.5-Lite full release: expert training, OPD distillation, one model at inference

Is there any interest in a self hosted open source version of manus/perplexity?

Make Jensen Huang Sound Like Anyone. New Streaming Voice Conversion Model MeanVC2 Released!

I did it! I'm free! It's been 7 hours since I used claudecode