r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 1d ago

zai-org/GLM-5.3 · Hugging Face

GLM-5.3 uses the same base model as GLM-5.2 — every gain comes from post-training. Compared with GLM-5.2, it is much better at complex coding and long-horizon tasks: Stronger Coding: GLM-5.3 is the most capable open-weights model for coding, with a 50% improvement over GLM-5.2…

15
r/LocalLLaMA community 1d ago

how to setup llama.cpp and blender to make lovely 3d stuff together

see here or read the following: ### I've found this workaround/complete setup: - git clone https://projects.blender.org/lab/blender_mcp.git - cd blender_mcp - uv --directory ./mcp/ run blender-mcp --transport http --port 9191 (this last command raise the error relative to this…

26
r/LocalLLaMA community 1d ago

Qwen3.8-27b q8 KV cache does seem to actually hurt model performance

EDIT: Though the issue with q8 kv cache seems to arise from when and how often we run the quantize step, not that kv quantizing can't ever work - see comments --- One of the things I see debated a lot is whether to use kv cache quantization. The idea I see a lot is that q8…

28
r/LocalLLaMA community 1d ago

Heat!

As in actual physical heat. How are people coping with the heat that running a decent inference rig pumps out? I've got dual 5060 Ti GPUs and a relatively modest CPU (Intel 14 Core Ultra 5 245KF Desktop on an Msi MPG Z890 motherboard) and if I use it for a coding session or…

8
r/LocalLLaMA community 1d ago

After Meta avocado we get watermelon, due in November

Quote: ... developing a new A.I. model intended to be as powerful as Anthropic’s cutting-edge models. ... In July, while developing Watermelon, Meta paused and later resumed a stage of A.I. development called “pretraining,” which delayed its release until at least October, four…

38
r/LocalLLaMA community 1d ago

Qwen3.8-Flash-Next (UD-IQ4_XS) on 2x RTX 3060 + 7800X3D, from initial 36 tps prefill to 400 tps and other benchmarks (-sm tensor trap) + VRAM/RAM usage

TL;DR: llama.cpp with --load-mode mmap used 21-32 GB of RAM, with ik_llama.cpp using 106-108 GB. -sm tensor killed my prefill, changing to -sm layer went from 36 tps to 135 tps. After that, -ubatch 2048 pushed it up to 400 tps at -c 131072 . I can't fit -ubatch 2048 at -c 262144…

24
r/LocalLLaMA community 2d ago

Micron: HBM Requires Three Times More Wafer Area Than DDR5

"At Hot Chips 2026, Micron drew a notable comparison: For the same memory capacity, HBM requires approximately three times the wafer area of DDR5." "When asked whether this ratio would improve with newer generations, the Micron Fellow reportedly explained that it definitely…

27
r/LocalLLaMA community 2d ago

TontaubeV1 - Open TTS model release for local long-form generation

Hey everyone, I am the co-founder of Tontaube. My brother and I just released TontaubeV1, a 2.9B-parameter open-weight TTS model focused on expressive speech, long-form generation, and low-latency local inference. The model is primarily aimed at English and German and supports…

36
r/LocalLLaMA community 2d ago

claude mods didn't like that, somehow 🤷‍♀️

  submitted by   /u/peculiar-ragdoll [link]   [comments]

37
r/LocalLLaMA community 2d ago

open source caught up because it's open

Proof is in the method honestly. Closed model labs need to constantly reinvent the wheel to keep lead. Open source has a bunch of independent labs practically working somewhat together. Eventually when everyone is just releasing weights and papers on how they did it the closed…

23
r/LocalLLaMA community 2d ago

Tencent/Hy4-preview 770B-A49B weight dropped

  submitted by   /u/Beamsters [link]   [comments]

11
r/LocalLLaMA community 2d ago

Infinite procedurally generated walking simulator coded entirely by Ornith-1.5-35B-Q4_K_M on an 8 GB RTX 4060

I'm very impressed with Ornith-1.5. It's the best model for my setup. It coded this in opencode.   submitted by   /u/Special_Condition671 [link]   [comments]

4
r/LocalLLaMA community 2d ago

I reverse-engineered an NPU vendor's engine format (int8 weights stored as two nibble planes) to run GGUFs with no model conversion — now 1.5× faster than the vendor's own runtime

I've been running Qwen3-0.6B on the M5Stack LLM-8850 card (Axera AX8850 NPU, 24 TOPS, 8GB LPDDR4x) hosted by a Raspberry Pi 5 — as a llama.cpp backend. The problem: the vendor stack requires converting every model through their compiler, and their closed runtime gets 13.5–14.5…

28
r/LocalLLaMA community 2d ago

Ornith 1.5 is actually pretty good

hey guys i recently started using ornith 1.5 to rapidly test some tools im working on since qwen 3.8 27b was too slow for my testing loop. This model is actually really good. im getting around 130 tokens / second with mtp and its very good at tool calling. i feel like this is…

32
r/LocalLLaMA community 2d ago

Ninfer and a 5090 with 3.8 27B is making me cry tears of joy it's so good.

Built the latest and I'm getting as much as 220 tokens per second and averaging in the 170s, I can't get over it. If anyone on here is on that project, fuckkkin' chapeau man, really incredible job. I can't believe I was able to like double or more my throughput from llama.cpp…

34
r/LocalLLaMA community 2d ago

Ornith-1.5-35B-A3B on 8 GB VRAM: I think I've found my sweet spot

A few days ago I posted asking what people considered the best local model for an 8 GB VRAM GPU . At the time, my personal sweet spot was Qwen3.6-35B-A3B , for agentic coding with Pi.dev. Well… Thanks to the suggestions in that thread, I think I've found something even better.…

25
r/LocalLLaMA community 2d ago

I am Concerned if Nvidia Acquires Llama.CPP, Dev Team and HF, Anybody else?

I dont know about others, but Nvidia is aiming (potentially) to close the lid on older GPUs since they want to push their new technology. Llama and team has been the to go places for older GPUs like V100s. Knowing how Nvidia have tried killing these GPUs of relevancy concerns me…

25
r/LocalLLaMA community 2d ago

Do you think a few Qwen3.8-27B models working together could score as well as Fable-5 on LiveCodeBench Hard?

Has anyone tested this? Ensemble of small Qwen models claiming Fable 5-level coding performance. A new paper claims that running several Qwen3.8-27B models together matches Fable 5’s accuracy on LiveCodeBench. The authors also say their setup paired with GPT Terra reaches Fable…

11
r/LocalLLaMA community 2d ago

yall are sleeping on qwen 3.8 27b q2 + q2 dflash + q5 kv

ok bit more context: it's actually a QAT Q2 for Qwen 3.8 27 B: https://huggingface.co/sdkyuan/qwen3.8-27B-qat-q2_0-gguf QAT Q2 for DFlash model: https://huggingface.co/HermiHg/Qwen3.8-27B-DFlash2-Q2_K_S-MIX-GGUF Q5 KV seems to cause 0 problems for me; I've used it up to 200K…

36
r/LocalLLaMA community 2d ago

I implemented a modern LLM in 700 lines of C

I’ve been working on a small project called gemma4.c. The idea is pretty simple: you can download a modern language model, compile one 700-line C file, and have it generate text on an ordinary CPU. Then you can read that same file from top to bottom and understand exactly how…

33
r/LocalLLaMA community 2d ago

Appreciation Post - thomsonreuters/Thomson-1.0-Small

With the lack of support from Qwen regarding the smaller 9B and 35B MOE models. Like myself, not everyone is looking for an agentic coding model, I particularly use it for RAG and reviewing and require high reasoning across different documents & came across this Finetune:…

22
r/LocalLLaMA community 2d ago

Agent Arena Code - Very good result (preliminary) for GLM and Qwen!

Everything has changed in two months: DS4 0731 flash was the start of a wave that is taking open weights to paradise. It is easy to think that Qwen 4 and GLM 6 will be on par with Mythos.   submitted by   /u/LegacyRemaster [link]   [comments]

22
r/LocalLLaMA community 2d ago

5090 now officially cost 5090

I was planning on another 5090, but then I realize... perhaps I am much better off getting an M5 Ultra Mac Studio with 256gb of ram. We are so genuinely cooked.   submitted by   /u/Sadge404 [link]   [comments]

5
r/LocalLLaMA community 2d ago

The Unsloth appreciation post. BIG thanks to Daniel and Michael! Thanks from the community to you guys for so much!

With HF being bought out and its future feeling a little iffy, I got to thinking about the teams that have constantly looked out for the little guys and stayed true to their open-source roots. There are great developers who share their work freely, and the local AI scene is…

35
r/LocalLLaMA community 2d ago

Let’s be real, memory and gpus price will continue go up next year and the year

Models will continue improve for open and closed labs, the demand for compute and memory will continue to increase. Expect to pay double or more for ddr 5 and 6 ram and +60% plus for new consumer gpus . Even a 512 gb mac studio will likely be over 22k .   submitted by  …

8
r/LocalLLaMA community 2d ago

Over 200k context on 16GB VRAM with Qwen 3.8 27B UD-IQ3_XXS

I was using UD-Q3_K_XL until now with more than 140000 context. Quality wise it's very good, very few erroneous tool calls. Then I saw many others here reporting good results with IQ3_XXS, so I gave it a try. The downside is prompt processing speed went down from 700-800 tk/s to…

7
r/LocalLLaMA community 2d ago

llama.cpp support for Qwen3.8-Flash-Next has been merged

finally I can download the GGUF   submitted by   /u/jacek2023 [link]   [comments]

23
r/LocalLLaMA community 2d ago

Bill Gates is looking to meet with Chinese President Xi Jinping later this year, eager to propose global efforts to mitigate the growing risks posed by artificial intelligence. He believes China might agree to restricting potentially dangerous AI model releases if US took…

  submitted by   /u/f0urxio [link]   [comments]

38
r/LocalLLaMA community 2d ago

With HuggingFace, Nvidia is also acquiring llama.cpp and the team behind it

With this move Nvidia is not only acquiring the HuggingFace platform, but they might also effectively acquire the copyright to the `llama.cpp` project, together with the entire team behind it. In February 2026 the llama.cpp team was employed by HF in order to continue working on…

37
r/LocalLLaMA community 2d ago

No, Engrams won't let you run 1T models locally. It does something even better.

Ever since Qwen 3.8 Flash Next dropped, there's a misconception going around that N-gram tables will let people run 1T+ parameter models on a single server with 980B parameters offloaded to SSD. I'm here to disappoint you: it won't. But what it will actually do for local models…

31
r/LocalLLaMA community 2d ago

[audio.cpp] Release 0.7: 62 audio model families (85+ variants), Arena UI for model comparison, MiniMax Music 3, FireRed TTS3/Audio, ControlFoley, Personaplex, and more

audio.cpp 0.7 is out :) This release adds a lot of new audio models and a new way to compare them locally. Audio.cpp is now at 62 model families and 85+ model variants. And it keeps growing! The biggest user-facing change is the new Arena UI . Instead of testing one model at a…

22
r/LocalLLaMA community 2d ago

Support for DFlash2 in llama.cpp has been merged! - spec : add DFlash2 support (local convolution + candidate selector) by SubSir · Pull Request #27342 · ggml-org/llama.cpp

  submitted by   /u/DjCanalex [link]   [comments]

14
r/LocalLLaMA community 2d ago

NVIDIA Next Gen Vera Rubin GPUs scheduled for mid-2027

Here’s hoping RTX 6090 also comes at MSRP of $6969 lol. With how expensive RTX 5090s and 6000 Pros have gotten it doesn’t sound too far fetched.   submitted by   /u/Leafytreedev [link]   [comments]

13
r/LocalLLaMA community 2d ago

I used local Qwen 27b to build a harness and replace OpenCode

Sharing my harness for running local LLMs that I built using Qwen 3.x 27B (> 90% locally built) under my supervision - not vibe-coded. Its free, no telemetry, and open-source (AGPL). Works on Windows, Linux (sorry, no Mac yet). I use it for my own coding + mixed workflows. How…

13
r/LocalLLaMA community 2d ago

We’re the Team Behind Apodex 1.1 — Ask Us Anything!

Hi r/LocalLLaMA ! We’re Apodex , the team behind Apodex 1.1 , our new model family built to scale agentic intelligence for complex work. We’re excited to be here and answer your questions directly. Apodex 1.1 is designed around sustained, verifiable progress toward real-world…

18
r/LocalLLaMA community 2d ago

Request: unsloth Please re-quantize Qwen3.6 35 A3B and 27B using UD 3.0

UD 3.0 seems to be a massive improvement over UD 2.0 Some of us still want to run the older Qwen models but would benefit from UD 3.0 UD 2.0 vs 3.0 is like the difference between a full quant. So Q3 UD 3.0 is similar to Q4 UD 2.0.   submitted by   /u/Fancy-Snow7 [link]…

14
r/LocalLLaMA community 2d ago

GLM-5.3 Flash Unsloth GGUF now available

  submitted by   /u/ElementNumber6 [link]   [comments]

30
r/LocalLLaMA community 2d ago

Microduck by Pollen Robotics & Hugging Face

Pollen Robotics and Hugging Face are releasing an open-source bipedal robot that comes with reinforcement learning software. It looks like it has a speaker, camera + LiDAR, NFC, Wifi, Bluetooth, etc. And roller-skates, because that's just the cutest thing ever. They announced it…

19
r/LocalLLaMA community 2d ago

llama: model_loader: add TENSOR_READ_LAZY by ngxson · Pull Request #27794 · ggml-org/llama.cpp

Qwen 3.8 Next Flash (Qwen 4) engrams don't need to be in VRAM/RAM   submitted by   /u/jacek2023 [link]   [comments]

8
r/LocalLLaMA community 2d ago

and then they came for the used server RAM.

I don't know why but when watching a video about FreeToken this morning this just came to mind lol.   submitted by   /u/MammothUnique4147 [link]   [comments]

35
r/LocalLLaMA community 2d ago

GLM-5.3-Flash @ DGX Station GB300: ~206 tok/s (single stream), 1M context

Hey all! I'm finally doing some cool stuff with my "thinking heater" (h/t u/-TV-Stand- ). I'm still experimenting with GLM-5.2 (in anticipation of 5.3 coming tomorrow, I hope!) and things are very cool so far. With the release of GLM-5.3-flash, I decided to play with it on the…

25
r/LocalLLaMA community 2d ago

friendly reminder you can legally torrent ai models.

Repost because reddit keeps thinking this is piracy or illegal. It is neither. A lot of people are skeptical Nvidia will keep huggingface intact now that they will buy huggingface. There's a lot of doom and gloom about not having any alternatives, removing nsfw models, saying…

21
r/LocalLLaMA community 2d ago

Qwen3.8-Flash-Next: Time to Update Those Benchmarks

specs hardware: M4 Max 128GB Studio inference engine: oMLX & lllama.cpp insights it still very early, so had to disable oMLX K/V caching, qwen4_exp architectureis not yet supported + the obvious n-grams with which the whole 4 bit quant takes ~100G, so pretty tight nevertheless,…

35
r/LocalLLaMA community 3d ago

Moderation !== Censorship

All this mega threads and censoring posts killing LocalLLaMA vibe. And yeah, I liked more when we had 20+ posts about new model.   submitted by   /u/inkberk [link]   [comments]

20
r/LocalLLaMA community 3d ago

What are the minimum specs required to run Qwen3.8-Flash-Next?

How much system RAM? How much VRAM? How much SSD space? Ideally list for q3/4 but q2 might also work since I have seen 3.8 27B perform well even on q2. Currently I have 5070 Ti with 16GB VRAM and 48GB system RAM. I can upgrade system RAM to 96GB is that will allow it to run.…

19
r/LocalLLaMA community 3d ago

Nvidia is buying Huggingface for $12.9 billion

  submitted by   /u/doomdayx [link]   [comments]

6
r/LocalLLaMA community 3d ago

llama : add --n-cpu-ffn option by John-194 · Pull Request #26622 · ggml-org/llama.cpp

tl;dr faster dense models for low VRAM people option similar to the existing --n-cpu-moe It puts user specified amount of FFN sublayers for dense models. PR by u/Stainless-Bacon   submitted by   /u/jacek2023 [link]   [comments]

7
r/LocalLLaMA community 3d ago

Qwen3.8-Flash-Next better then DeepSeek V4 Pro

  submitted by   /u/Normal-Phone7762 [link]   [comments]

30
r/LocalLLaMA community 3d ago

Dflash 2 speculative decoding by SamuelOliveirads · Pull Request #2345 · ikawrakow/ik_llama.cpp

Didn't update on ik_llama.cpp updates for sometime. Updates from this month: cuda : repair the HIP build, and validate IQ4_KS and IQ4_KT on RDNA3- #2339 vulkan : add IQ4_KS and IQ4_KT support- #2332 Initial implementation of DSpark model: Ling-3.0 (bailingmoe3) runtime support…

29
r/LocalLLaMA community 3d ago

NVIDIA buying HF isn't a good thing for open source

  submitted by   /u/johnnyApplePRNG [link]   [comments]

18

zai-org/GLM-5.3 · Hugging Face

how to setup llama.cpp and blender to make lovely 3d stuff together

Qwen3.8-27b q8 KV cache does seem to actually hurt model performance

Heat!

After Meta avocado we get watermelon, due in November

Qwen3.8-Flash-Next (UD-IQ4_XS) on 2x RTX 3060 + 7800X3D, from initial 36 tps prefill to 400 tps and other benchmarks (-sm tensor trap) + VRAM/RAM usage

Micron: HBM Requires Three Times More Wafer Area Than DDR5

TontaubeV1 - Open TTS model release for local long-form generation

claude mods didn't like that, somehow 🤷‍♀️

open source caught up because it's open

Tencent/Hy4-preview 770B-A49B weight dropped

Infinite procedurally generated walking simulator coded entirely by Ornith-1.5-35B-Q4_K_M on an 8 GB RTX 4060

I reverse-engineered an NPU vendor's engine format (int8 weights stored as two nibble planes) to run GGUFs with no model conversion — now 1.5× faster than the vendor's own runtime

Ornith 1.5 is actually pretty good

Ninfer and a 5090 with 3.8 27B is making me cry tears of joy it's so good.

Ornith-1.5-35B-A3B on 8 GB VRAM: I think I've found my sweet spot

I am Concerned if Nvidia Acquires Llama.CPP, Dev Team and HF, Anybody else?

Do you think a few Qwen3.8-27B models working together could score as well as Fable-5 on LiveCodeBench Hard?

yall are sleeping on qwen 3.8 27b q2 + q2 dflash + q5 kv

I implemented a modern LLM in 700 lines of C

Appreciation Post - thomsonreuters/Thomson-1.0-Small

Agent Arena Code - Very good result (preliminary) for GLM and Qwen!

5090 now officially cost 5090

The Unsloth appreciation post. BIG thanks to Daniel and Michael! Thanks from the community to you guys for so much!

Let’s be real, memory and gpus price will continue go up next year and the year

Over 200k context on 16GB VRAM with Qwen 3.8 27B UD-IQ3_XXS

llama.cpp support for Qwen3.8-Flash-Next has been merged

Bill Gates is looking to meet with Chinese President Xi Jinping later this year, eager to propose global efforts to mitigate the growing risks posed by artificial intelligence. He believes China might agree to restricting potentially dangerous AI model releases if US took…

With HuggingFace, Nvidia is also acquiring llama.cpp and the team behind it

No, Engrams won't let you run 1T models locally. It does something even better.

[audio.cpp] Release 0.7: 62 audio model families (85+ variants), Arena UI for model comparison, MiniMax Music 3, FireRed TTS3/Audio, ControlFoley, Personaplex, and more

Support for DFlash2 in llama.cpp has been merged! - spec : add DFlash2 support (local convolution + candidate selector) by SubSir · Pull Request #27342 · ggml-org/llama.cpp

NVIDIA Next Gen Vera Rubin GPUs scheduled for mid-2027

I used local Qwen 27b to build a harness and replace OpenCode

We’re the Team Behind Apodex 1.1 — Ask Us Anything!

Request: unsloth Please re-quantize Qwen3.6 35 A3B and 27B using UD 3.0

GLM-5.3 Flash Unsloth GGUF now available

Microduck by Pollen Robotics & Hugging Face

llama: model_loader: add TENSOR_READ_LAZY by ngxson · Pull Request #27794 · ggml-org/llama.cpp

and then they came for the used server RAM.

GLM-5.3-Flash @ DGX Station GB300: ~206 tok/s (single stream), 1M context

friendly reminder you can legally torrent ai models.

Qwen3.8-Flash-Next: Time to Update Those Benchmarks

Moderation !== Censorship

What are the minimum specs required to run Qwen3.8-Flash-Next?

Nvidia is buying Huggingface for $12.9 billion

llama : add --n-cpu-ffn option by John-194 · Pull Request #26622 · ggml-org/llama.cpp

Qwen3.8-Flash-Next better then DeepSeek V4 Pro

Dflash 2 speculative decoding by SamuelOliveirads · Pull Request #2345 · ikawrakow/ik_llama.cpp

NVIDIA buying HF isn't a good thing for open source