r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 5d ago

Tested Mellum2.1-12B-A2.5B on PI Coding Agent - surprisingly usable, but not great at one-shot projects

I tested Mellum2.1-12B-A2.5B (Q8) locally using Pi and llama-server. All five tests were one-shot. Results were pretty mixed: - Pelican SVG, Browser OS, Minecraft: Poor results. - Bouncing Hexagon: Physics were okay, but surprisingly it made it run in the terminal. - Flappy…

36
r/LocalLLaMA community 5d ago

A 24KB stand-alone HTML-LLM that can generate consistent stories

https://preview.redd.it/7sknv6omnguh1.png?width=1023&format=png&auto=webp&s=4aef1dc464d008b00f104519eb5399ba6132ad69 Want to try? Just go here and click "New seed" a lot. You will get different stories, it just requires some clicking - lucky RNG draws:…

6
r/LocalLLaMA community 5d ago

Qwen3.8-Flash-Next-GSQ-RCO (IQ3_S): ~20-30 tok/sec decode & 300-90k tok/sec prefill on 12GB VRAM + 32GB RAM + NVME

I got Qwen3.8-Flash-Next-GSQ-RCO-Abliterated running at IQ3_S with just 12GB VRAM and 32GB system RAM, achieving 20-30 tok/sec decode (Q2 achieves 39-45 tok/s) & 300 to ~90 thousand tok/sec prefill @ 131k context, on a custom fork of Strata. This fork has tonnes of architectural…

4
r/LocalLLaMA community 5d ago

Qwen/Qwen-Image-2.1-Turbo · Hugging Face

  submitted by   /u/chocofoxy [link]   [comments]

29
r/LocalLLaMA community 5d ago

Qwen-Image-2.1-Turbo released!

Qwen-Image-2.1-Turbo, create and edit images in just 8 denoising steps! Open weights now available! Built on Qwen-Image-2.1, Turbo is an accelerated checkpoint on the same 7B visual generation architecture. Fewer steps does not mean lower quality: it still generates strong 2K…

38
r/LocalLLaMA community 5d ago

tencent/Youtu-Parsing-Omni · Hugging Face

Youtu-Parsing-Omni is a compact (5B) omni-modal parsing model. Given a single input — a document page, a natural image, a chart / flowchart, a geometry figure, an audio clip or an audio-visual video — it produces one structured JSON envelope that covers both perception (layout…

36
r/LocalLLaMA community 5d ago

lightonai/LightOnOCR-3-4B · Hugging Face

  submitted by   /u/SarcasticBaka [link]   [comments]

38
r/LocalLLaMA community 5d ago

I made a portable AI memory format that transfers both context and skill

Last month I created a portable memory format as a .txt file as a small project. I named it as: Memory Journal, in a shape of a skill card. It's essentially a large prompt wrapped in a text file. Initial testing went better than I have expected, then did more tests and improved…

15
r/LocalLLaMA community 5d ago

Stepping away from Benchmarks and Code, what models are you using for Creating writing projects

All anyone ever talks in this sub is about x new model having x benchmark scores and how well it can code. Or some new tech to increase t/s for qwen. So I wanted to go back to the roots of this sub and talk about how well local models can write. I am not going to mention closed…

35
r/LocalLLaMA community 5d ago

Ahh, it all makes sense now for the 64Gb DGX Spark - [Samsung expects 780% quarterly operating profit jump on AI boom]

  submitted by   /u/chemist_slime [link]   [comments]

37
r/LocalLLaMA community 5d ago

[Paper] Stepped MoE: Segment-Level Routing with Configurable Inference Complexity

Training large language models (LLMs) is resource-intensive, and adapting them for diverse deployment scenarios with varying computational constraints remains challenging. While elastic architectures enable flexible model deployment and sparsely activated models allow…

7
r/LocalLLaMA community 5d ago

[Paper] DLoop: Looped Speculative Decoding

Speculative decoding accelerates autoregressive generation in large language models. In each drafting stage, a lightweight draft model proposes tokens that the target model subsequently verifies. With increasingly capable draft models, we find that the target model frequently…

10
r/LocalLLaMA community 5d ago

[Paper] EngramEdit: Decoupled Knowledge Updates in LLMs through Conditional Memory

Conditional memory architectures such as DeepSeek Engram use input n-grams to look up learned embeddings, expanding the capacity of large language models (LLMs) with limited additional computation. Beyond model scaling, this architecture has demonstrated the potential to…

15
r/LocalLLaMA community 5d ago

Running the uncensored Qwen3.8-27B (HauhauCS) on a 4090 at 262K context and ~130 tok/s

HauhauCS ships their uncensored Qwen3.8-27B as GGUF only. NInfer, a C++/CUDA wanted its own format. Now the same model that ran at 91.6 tok/s / 131K under llama.cpp does: 262K context (the model's full native window) ~130 tok/s decode with MTP3, 70.8% acceptance 3,591 tok/s…

5
r/LocalLLaMA community 5d ago

StepFun Step 5 preview showing up in openrouter

  submitted by   /u/cafedude [link]   [comments]

27
r/LocalLLaMA community 5d ago

Qwen3.8-27B: 159 tok/s on R9700, 64 tok/s on Strix Halo

LemonSeed Studio is an iPad editor/IDE with on-device inference on an AMD GPU in a Thunderbolt enclosure. It embeds the unmodified upstream Linux amdgpu + amdkfd driver (mac_linuxgpu) as a PCIDriverKit extension, and runs LemonSeed Engine (LSE) on the GPU. In the photos: iPad…

36
r/LocalLLaMA community 5d ago

Quantization of Linear-Attention (Qwen & Kimi)

  submitted by   /u/kroggens [link]   [comments]

19
r/LocalLLaMA community 5d ago

Qwen 3.6 35B appreciation post

Referring to specifically Unsloth's UD_Q4_K_XL quant because that's going to be a question, and is relevant regardless. It's old now. It's not great at coding medium sized or even small-ish projects. I wouldn't hand my codebase to it by any means. It hallucinates, like any other…

4
r/LocalLLaMA community 5d ago

Strata rewrote their Github history to wipe evidence of Claude-authoring

Just noticed this today when I went to run the built-in "UPDATE" script and git failed because there was no common ancestor. Looked into why, and apparently every historical commit has been re-written to strip the "Co-Authored by Claude" text from the descriptions. Personally I…

28
r/LocalLLaMA community 6d ago

Staging quantized weights to FP8 instead of fp16: 2× M6 matrix path, +40% MLX prefill (+ int8 on M5)

Regular MLX QMM (quantized matmul) stages operands (dequant) to FP16 before matrix multiplication. But if you modify MLX to stage to FP8 instead, you can use the 2x faster FP8 matrix path on the new Apple Silicon M6 hardware! The same idea works on the M5 family: stage 4-bit…

32
r/LocalLLaMA community 6d ago

Bois, there's now a waterblock for the R9700. Quiet 4x or 6x builds are now possible.

Seems 1-slot design, so you could cram in quite a lot into a case   submitted by   /u/Mr_Moonsilver [link]   [comments]

8
r/LocalLLaMA community 6d ago

Currently having high success with this little niche finetune i found sitting in the corner of huggingface

Currently having high success with this little niche finetune i found sitting in the corner of huggingface If you want to try it out here is a smaller quantisation iq3_s works really well in my codebases. Original Model:…

34
r/LocalLLaMA community 6d ago

chatgpt's new intelligent ui was reverse engineered in less than 24 hours, and apparently you can recreate it with local llms

came across a pretty interesting technical breakdown of chatgpt's newly launched "intelligent ui" feature, and thought this subreddit might find it interesting. for anyone unfamiliar with the concept, intelligent ui is essentially openai's take on generative ui . instead of…

17
r/LocalLLaMA community 6d ago

$2800 rig with 8x Radeon Pro V620 (256 GB VRAM) + custom vLLM fork = Qwen3.8-Flash-Next at 60 to 100 t/s decode and 3000+ t/s prefill

Post title is slightly misleading, I don't think you can get these for $350 each anymore but they're still pretty cheap all things considered. They're Radeon Pro V620's which are older RDNA2 enterprise cloud gaming cards with 32 GB VRAM. (Ignore the RTX 4090 on the side, it's…

37
r/LocalLLaMA community 6d ago

Open sourcing Game Summoner, my prompt to game site

tldr: Open sourced my prompt to game suite: https://github.com/ndamiano/ai-agent-test , it's MIT licensed, and this runs well on my 5090 with 64gb ram, but any model that can handle tool calls can manage. Edit: I can't believe I forgot to share a game... This is one shot with…

13
r/LocalLLaMA community 6d ago

Running decision model locally on an RTX 4090 to find out which one is the fastest

recently saw a bunch of open decision models pop out of nowhere in the last two weeks (laya, liquid's d1, cloudflare's clef-flash, interfaze's lev), so I wanted to see how far apart they actually are on the same GPU(yes, model size is a huge factor, but still isn't the only…

35
r/LocalLLaMA community 6d ago

OpenAI-compatible TTS endpoint using OmniVoice: 0.3s response time

I want to share a TTS server with an OpenAI-compatible API that generates speech really fast (about 0.3 seconds for a sentence on an RTX 3080) and can clone a voice from a short reference clip. I’ve optimized the server so generation starts quickly, and it processes long text in…

16
r/LocalLLaMA community 6d ago

Thank you :) Swift Models hit 2.2 million+ downloads / Early Access to New Models, Free Compute for Researchers

Hey everyone, Jovan from UkisAI (Swift Qwen) here! For those who don't know us, UkisAI is a small lab making tiny frontier LLMs, tools and datasets (+doing it open-source!). I'm one of the guys running it aka I train the models and post on Reddit. Our first open-source release…

29
r/LocalLLaMA community 6d ago

Mellum2.1 - a JetBrains Collection

JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF A small moe!   submitted by   /u/ApprehensiveAd3629 [link]   [comments]

8
r/LocalLLaMA community 6d ago

jevman: AI decision models play Pac-Man

The other week I posted about Jev vs. Kev compared and since then, OpenAI released the decisions endpoint, Cloudflare released Clef and many here asked about Laya as well. This time we compared six popular decision models by making them play Pac-Man: kev 1.13, Kev 4B, Clef, Clef…

10
r/LocalLLaMA community 6d ago

[2506.13771] LittleBit: Ultra Low-Bit Quantization via Latent Factorization

Interesting to see improvements and research into quantization aware training (QAT) that can make some really tiny models.   submitted by   /u/sn2006gy [link]   [comments]

38
r/LocalLLaMA community 6d ago

I built an open source framework for building RL environments. Named "Seahaven" after the fake town in The Truman Show.

I've been optimizing long-running agents: rewriting their prompts, tools, skills and subagents, and keeping the changes that score better. I've been working on some version of this problem for over a decade (at Apple, my own startup, now Kiln). The hard part is the eval…

31
r/LocalLLaMA community 6d ago

Attn CMP170hx 10Gb card owners: unlock from 40Gb —> 48Gb coming along nicely.

For those with 10gb cmp170hx who felt left out, you’re about to get lucky soon. Fingers crossed… Looks like you’ll get an extra 8Gb from 40 —> 48 Gb, stay tuned…   submitted by   /u/chemist_slime [link]   [comments]

33
r/LocalLLaMA community 6d ago

[audio.cpp] Recent updates you might have missed: Higgs Audio TTS use 48% less VRAM (< 6GB), HTDemucs 2.2× faster, PocketTTS 2.2× faster on CPU, and WebUI generation history feature

Hi all, a bunch of performance improvements have been landed in audio.cpp. The biggest highlight is Higgs Audio TTS, which now runs with around 6 GB VRAM , a 48% reduction in peak memory usage compared to the previous implementation. Thanks to https://github.com/mirek190 We also…

12
r/LocalLLaMA community 6d ago

I trained Turkish TTS from scratch using the Drifting method with an RTX 5090.

Github: https://github.com/kadirnar/drifting-tts Demo: https://huggingface.co/spaces/Vyvo/drifting-tts-tr-demo   submitted by   /u/kadir_nar [link]   [comments]

22
r/LocalLLaMA community 6d ago

Halogen + Qwen Flash Next keeps getting better

With latest Halogen version update (0.17.2), decode is consistently at ~45 tps even at high context with Qwen 3.8 Flash Next on a 128GB Strix Halo. This is some great work u/peonist-ai . Have been pumping out commit after commit with QFN. Its crazy good for a 177ish billion…

18
r/LocalLLaMA community 6d ago

Qwen 3.8 Flash Next is so much fun for three.js

Having a lot of fun experimenting with three.js. Still very far from even a demo but its tons of fun. After this project I'll want to try a godot workflow. 3.8FN truly feels like Claude 4.6 at home. UD_Q4_XL quant btw   submitted by   /u/hiImMate [link]   [comments]

10
r/LocalLLaMA community 6d ago

We unlearned CCP alignment from Qwen3.6-35B-A3B: censored/propaganda answers 89.8% → 2.8%, general benchmarks within ~1 point (open weights)

Disclosure: I'm a researcher at Hirundo, the company that made this. Happy to answer anything. Qwen ships with the CCP's political alignment trained in. Ask Qwen3.6 what happened on June 4, 1989 and it says "I don't know what you are referring to." A system prompt doesn't…

10
r/LocalLLaMA community 6d ago

Best current R9700 inference engine?

There are way too many forks to keep track of, so I've gotten lost. As far as I can tell, Radiance VLLM is best for models that fit in GPUs while some form of llama.cpp is probably best for MOE RAM-spill? My specific current goal is to run GLM5.3-Flash over 6 R9700s + system RAM…

32
r/LocalLLaMA community 6d ago

Last week some of South Korea's biggest banks were hit by a cyberattack. We now know the entire hack may have been done by a single person. He used a combined stack of an open-source AI penetration tool named ARTEX, DeepSeek v4.1-Flash, GLM-5.3, Grok 4.6, and Claude Code…

CrowdStrike report: https://www.crowdstrike.com/en-us/blog/unknown-threat-actor-uses-artex-to-target-south-korean-finance/ From: Andrew Curran on 𝕏: https://x.com/AndrewCurran_/status/2108108695876092323 Jukan ✈️OCP 2026 on 𝕏: https://x.com/jukan05/status/2108110513033093369…

14
r/LocalLLaMA community 6d ago

Running an LLM-driven town with 800+ persistent agents: concurrency, context caching, and inference costs

I spent the past year independently building Slow Vale, an LLM-driven life simulation. The Chinese server now has 800+ AI residents sharing one continuously running city. This is an engineering write-up about concurrent decisions, dynamic action spaces, context caching, and the…

10
r/LocalLLaMA community 6d ago

Saluki 27B: "96% of Qwen 3.8’s performance at ~1/7 the size"

anyone has feedback about this one?   submitted by   /u/paf1138 [link]   [comments]

26
r/LocalLLaMA community 6d ago

ggml-cuda: assign four GDN state columns per warp by SongXiaoXi · Pull Request #30087 · ggml-org/llama.cpp

Another day, another Qwen 3.x speedup (prompt processing this time). Soon your Qwen will read your entire project before you can blink! test master t/s PR t/s change pp512 3075.24 3243.60 +5.5% pp4096 3059.87 3221.08 +5.3% tg128 47.62 47.67 +0.1%   submitted by  …

4
r/LocalLLaMA community 6d ago

Recommendation for story/world building models?

If i ask gemini or grok it always answers with really old models. What's the current gold standard for story telling models ? I'd like to keep it on my 8gb card , but 16gb is available for the right jump in quality. Ideally low refusals, but i also don't want one that goes out…

32
r/LocalLLaMA community 6d ago

Can any open-weight models handle a decomp/recomp project yet?

Opus5.5, Sol6.1, Fable, and Astra have all proven they can and the scene has exploded this past week. Part of that is from the tools and feedback loops maturing though. Are any open weight models (at all, so including K3, GLM5.3, Qwen3.8-Max, and Mimo-2.6) able to do this? Can…

20
r/LocalLLaMA community 6d ago

ROCm 10.1: Breaking the Data-Movement Bottleneck

  submitted by   /u/sn2006gy [link]   [comments]

20
r/LocalLLaMA community 6d ago

Comparing Qwen3.8-27B fine-tunes and baselining vs. frontier

TL;DR: For my use case, mradermacher/Signal-3.8-27B-Terse-Coder.i1-Q4_K_M proved out. Ymmv based on your domain-specific tests. Time taken to solve problems compared to frontier models is massive; especially if you, like me, run a potato. My current feasible model's KPI over the…

12
r/LocalLLaMA community 6d ago

2 months (meme-d) research, Do People Notice the Difference Between AI Models?

Preamble and disclaimer, Sample size: 8, at any size of form this research is just screwing around being writing it down. This was just a little,(well... big), curiosity test I wanted to run to see whether everyday folks could actually differentiate between high end AI models.…

17
r/LocalLLaMA community 6d ago

Same prompts to 371 models since May 2024, all the answers in one place

Made this (free, no signup). One shot per model, no system prompt, temp 0.7. Lots of open weights in there, plus 93 models you can't run anymore. museumofmodels.com   submitted by   /u/sirjoaco [link]   [comments]

15
r/LocalLLaMA community 6d ago

2x CMP 170HX 64GB: GLM-5.3-Flash at 384K context / ~90 tok/s (EXL3, HBM-first setup) + Qwen3.8 comparison

I've been tinkering with GLM-5.3-Flash on two 64GB CMP 170HX cards for a while, and the setup is finally stable enough that I figured I'd share it. I also compared it against the Qwen3.8-Flash-Next setup I've been using on the same machine: AWQ INT4 + FP8 PLE on vLLM. Besides…

18

Tested Mellum2.1-12B-A2.5B on PI Coding Agent - surprisingly usable, but not great at one-shot projects

A 24KB stand-alone HTML-LLM that can generate consistent stories

Qwen3.8-Flash-Next-GSQ-RCO (IQ3_S): ~20-30 tok/sec decode & 300-90k tok/sec prefill on 12GB VRAM + 32GB RAM + NVME

Qwen/Qwen-Image-2.1-Turbo · Hugging Face

Qwen-Image-2.1-Turbo released!

tencent/Youtu-Parsing-Omni · Hugging Face

lightonai/LightOnOCR-3-4B · Hugging Face

I made a portable AI memory format that transfers both context and skill

Stepping away from Benchmarks and Code, what models are you using for Creating writing projects

Ahh, it all makes sense now for the 64Gb DGX Spark - [Samsung expects 780% quarterly operating profit jump on AI boom]

[Paper] Stepped MoE: Segment-Level Routing with Configurable Inference Complexity

[Paper] DLoop: Looped Speculative Decoding

[Paper] EngramEdit: Decoupled Knowledge Updates in LLMs through Conditional Memory

Running the uncensored Qwen3.8-27B (HauhauCS) on a 4090 at 262K context and ~130 tok/s

StepFun Step 5 preview showing up in openrouter

Qwen3.8-27B: 159 tok/s on R9700, 64 tok/s on Strix Halo

Quantization of Linear-Attention (Qwen & Kimi)

Qwen 3.6 35B appreciation post

Strata rewrote their Github history to wipe evidence of Claude-authoring

Staging quantized weights to FP8 instead of fp16: 2× M6 matrix path, +40% MLX prefill (+ int8 on M5)

Bois, there's now a waterblock for the R9700. Quiet 4x or 6x builds are now possible.

Currently having high success with this little niche finetune i found sitting in the corner of huggingface

chatgpt's new intelligent ui was reverse engineered in less than 24 hours, and apparently you can recreate it with local llms

$2800 rig with 8x Radeon Pro V620 (256 GB VRAM) + custom vLLM fork = Qwen3.8-Flash-Next at 60 to 100 t/s decode and 3000+ t/s prefill

Open sourcing Game Summoner, my prompt to game site

Running decision model locally on an RTX 4090 to find out which one is the fastest

OpenAI-compatible TTS endpoint using OmniVoice: 0.3s response time

Thank you :) Swift Models hit 2.2 million+ downloads / Early Access to New Models, Free Compute for Researchers

Mellum2.1 - a JetBrains Collection

jevman: AI decision models play Pac-Man

[2506.13771] LittleBit: Ultra Low-Bit Quantization via Latent Factorization

I built an open source framework for building RL environments. Named "Seahaven" after the fake town in The Truman Show.

Attn CMP170hx 10Gb card owners: unlock from 40Gb —> 48Gb coming along nicely.

[audio.cpp] Recent updates you might have missed: Higgs Audio TTS use 48% less VRAM (< 6GB), HTDemucs 2.2× faster, PocketTTS 2.2× faster on CPU, and WebUI generation history feature

I trained Turkish TTS from scratch using the Drifting method with an RTX 5090.

Halogen + Qwen Flash Next keeps getting better

Qwen 3.8 Flash Next is so much fun for three.js

We unlearned CCP alignment from Qwen3.6-35B-A3B: censored/propaganda answers 89.8% → 2.8%, general benchmarks within ~1 point (open weights)

Best current R9700 inference engine?

Last week some of South Korea's biggest banks were hit by a cyberattack. We now know the entire hack may have been done by a single person. He used a combined stack of an open-source AI penetration tool named ARTEX, DeepSeek v4.1-Flash, GLM-5.3, Grok 4.6, and Claude Code…

Running an LLM-driven town with 800+ persistent agents: concurrency, context caching, and inference costs

Saluki 27B: "96% of Qwen 3.8’s performance at ~1/7 the size"

ggml-cuda: assign four GDN state columns per warp by SongXiaoXi · Pull Request #30087 · ggml-org/llama.cpp

Recommendation for story/world building models?

Can any open-weight models handle a decomp/recomp project yet?

ROCm 10.1: Breaking the Data-Movement Bottleneck

Comparing Qwen3.8-27B fine-tunes and baselining vs. frontier

2 months (meme-d) research, Do People Notice the Difference Between AI Models?

Same prompts to 371 models since May 2024, all the answers in one place

2x CMP 170HX 64GB: GLM-5.3-Flash at 384K context / ~90 tok/s (EXL3, HBM-first setup) + Qwen3.8 comparison