r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 3d ago

GLM-5.3 weights will be released tomorrow

The promise has been fulfilled.   submitted by   /u/serige [link]   [comments]

5
r/LocalLLaMA community 3d ago

this is a friendly reminder you can legally seed ai models via torrenting.

A lot of people are skeptical Nvidia will keep huggingface intact now that they will buy huggingface. There's a lot of doom and gloom about not having any alternatives, removing nsfw models, saying there's no decentralized alternative or just not trusting what Nvidia might do…

38
r/LocalLLaMA community 3d ago

N-gram vs Experts explained

Since Qwen's dropped the Qwen4Exp architecture bomb that focus on offloading parameters to n-gram instead of pure mixture of experts, I dug into this and learned quite a lot. Here's the summary. Expect mistakes from human's writing lol. TLDR: MoEs do reasoning, N-grams do…

21
r/LocalLLaMA community 3d ago

Local LLM harness for reverse engineering software?

Does anyone know of a decent reverse engineering harness/workspace setup for local models that I can just point the model at and have it go to work until it's reversed most if not all of the functions in a binary, even if it takes days? Of course, I'm willing to put effort into…

13
r/LocalLLaMA community 3d ago

Nvidia has been in talks to acquire Hugging Face for more than $13 billion - Business Insider

https://www.businessinsider.com/nvidia-in-talks-to-buy-hugging-face-13-billion-dollars-2026-8 Edit: The Information is reporting that the deal is done: Nvidia Agrees to Buy Open Source Model Repository Hugging Face For $12.9 Billion (paywall):…

18
r/LocalLLaMA community 3d ago

Qwen 27b ud IQ3XXS potential- 3D Zen Room demo - pi harness - build deploy and share link on discord

  submitted by   /u/dreamai87 [link]   [comments]

16
r/LocalLLaMA community 3d ago

Qwen3.8 27B C8 at 972 TG / 5,680 PP on 4x MI100 rig ($6.5k) using my new INT8 vLLM fork

Yet another vLLM fork thread here, but this time its for older INT8-centric hardware. This is a complete INT8 serving stack for Qwen3.8 27B based on vLLM, AITER, and a 27B GPTQ INT8 quant w/ DFlash2 . Its not just another vibed autoresearch loop. No, vLLM ships with very little…

9
r/LocalLLaMA community 3d ago

How to Fine-Tune an LLM: An End-to-End Guide

I ended up fine tuning a mistral 7b to outperform our costly foundational model and saved $300k. I previously thought that fine tuning was pointless (it's definitely not) and that all these problems could be solved with RAG (they can't). The truth is, a LoRA/QLoRA adapter is…

23
r/LocalLLaMA community 3d ago

Any news about DeepSeek V4 Flash Vision weights?

I'd be curious to try it locally since I use 0731 daily but still no news on the weights   submitted by   /u/LegacyRemaster [link]   [comments]

12
r/LocalLLaMA community 3d ago

Compared Qwen 3.8 27B community quants on RTX 6000 vs Claude Opus 4.6

*part 2 of an earlier post: previous quant comparison with voxel island creation this time I rented three rtx pro 6000 96gb, on each one I launched a qwen 3.8 27b quant and gave them 4 identical prompts: classical pool game air hockey 1v1 battle foosball official match…

37
r/LocalLLaMA community 3d ago

Lemonade end-of-summer project update, now serving 15 engines!

Hi everyone, it's been a while since I posted so here's an update on what the Lemonade community has been up to this summer. Our overall mission is to enable local AI builders with everything they need to make great apps and agents, while keeping the stack turnkey, portable, and…

23
r/LocalLLaMA community 3d ago

Anyone else doing eGPUs (OCuLink)?

Upgraded to a 5070 Ti so I could run Qwen 3.8 27B, which works perfectly, but didn't want to let the old 4070 Ti go to waste. The cards would touch if I put them both in the PC and I knew the heat would be awful from my crypto mining days. I always was curious about eGPUs so I…

33
r/LocalLLaMA community 3d ago

Self-hosting LLMs on budget hardware: general principles, hardware, benchmarks and frontends

Hello, I've been self-hosting LLMs on various budget hardware for a while (6x RTX 3060 12 GB, Intel Arc Pro B60 24 GB, RX 9070 XT, etc). Over the last few months, I wrote about it in 4 articles: General principles Hardware and inference optimization CPU+RAM offloading, MoE,…

36
r/LocalLLaMA community 3d ago

Can we reconsider the megathreads?

In the past during model releases there used to be tons of interesting discussions happening on this subreddit. However, the new rules of forcing everything into a single megathread almost completely killed off the discussions as far as I can tell. I get that some people didn't…

11
r/LocalLLaMA community 3d ago

Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

I measured various Qwen3.8 27B quantizations by Unsloth on popular benchmarks: FPQA Diamond, IFBench, and Terminal-Bench-2.1. Q4_K_M is all you need.   submitted by   /u/pmigdal [link]   [comments]

4
r/LocalLLaMA community 3d ago

Are models with N-Gram tables going to completely change the AI race?

The news about Qwen 3.8 Flash Next is the first I'm reading about n-gram tables. I may be completely misunderstanding how they work but it seems they could open the door for 1T+ parameter models to be run on a single server with modest GPUs and a ton of system RAM rather than…

22
r/LocalLLaMA community 3d ago

little tool for offline wikipedia RAG

I was bored and handwrote a tiny 100-line bash script to let an agent search for and read articles from an offline wikipedia archive during a regular chat. It's not particularly useful, but it's definitely neat and a big step up from llama-cli. I got surprisingly good results…

7
r/LocalLLaMA community 3d ago

Whoever the fuck predicted we would have gpt 5.5 performance in coding on consumer hardware a couple months ago now, i applaud you

Like wtaf? Qwen 3.8 27b is crazy. Can't wait for kimi k3 performance   submitted by   /u/GrokiniGPT [link]   [comments]

21
r/LocalLLaMA community 3d ago

[Megathread] GLM-5.3-Flash - former ox-alpha

Megathread for discussing the release of GLM-5.3-Flash. Quants Fine-Tunes & Abliterations Chat Templates Inference Server Support & Configuration Experiences, Benchmarks & Model Comparisons We'll try to clean up future duplicates around the release and point them here.…

35
r/LocalLLaMA community 3d ago

What's your most reliable model, even if it's "outdated"?

What's a model you keep coming back to even though newer ones have technically surpassed it? I've noticed I default to the same one for daily tasks despite downloading every shiny new release. Curious if others have a reliable workhorse they trust over benchmark leaders  …

31
r/LocalLLaMA community 3d ago

Gemma4 31B vs Qwen3.8 27B - why the huge difference in benchmarks?

Hi all, I'm looking for the best model for a hobby project and trying to make sense of the various data I came across. I know benchmarks do not often translate to the real world, especially to your particular use case (whatever it may be). But this is truly baffling: AA says…

28
r/LocalLLaMA community 3d ago

HF exploring sale - impact on open models?

Hugging Face is exploring sale of the business valued at around $13 billion dollars. Actually I don't think we have any other repo source. Which has the mix of model weights, datasets and Spaces. Kaggle is there and other academic repos. But as far as reach, ease of use. HF…

27
r/LocalLLaMA community 3d ago

zai-org/GLM-5.3-Flash · Hugging Face

  submitted by   /u/coder543 [link]   [comments]

20
r/LocalLLaMA community 3d ago

GLM-5.3-Flash: Frontier Intelligence, Flash Cost

  submitted by   /u/BriguePalhaco [link]   [comments]

27
r/LocalLLaMA community 3d ago

A minecraft clone I fully vibecoded with Qwen3.8-27b Q4

I wanted to see just how capable Qwen3.8-27b is locally. I have a RTX 4090 and 96GB of RAM but the Q4 comfortably fits in the GPU with plenty of context, the few times I needed more than 130k context I just loaded it spilled into RAM and it's capable of not degrading even at…

5
r/LocalLLaMA community 3d ago

Forget the Pelican, it's Weevil-Time! / Benchmaxxing-Proof SVG and Vision Benchmark

The Artist: Qwen3.8-27B-UD-Q3\ K_XL, q8_0 caches, xhigh, temp 1.0, image-min-tokens 1024, froggeric template) I was screwing around with different Qwen3.8-27B quants and thought of this very simplistic but seemingly bechmaxxing resistant combined SVG and vision test. Just let…

4
r/LocalLLaMA community 4d ago

A 27b model beating latest frontier models was not on my 2026 bingo card

https://preview.redd.it/kbsqh6f7molh1.png?width=730&format=png&auto=webp&s=068dbea9a50be634a369d54d8b27b781d020fab3 My experience with Qwen 3.8 for agentic tasks has been phenomenal but I personally feel that 3.7 flash is more reliable for overall tasks.   submitted by  …

26
r/LocalLLaMA community 4d ago

OpenCode with Qwen3.8-27B for Small Games or Browsing the Web With 16GB VRAM

In the past, I have use llama.cpp, but I read that the exl3 quantization format should give better precision , so I have tried exllamav3/tabbyAPI. It was able to write the shown simple HTML game without interaction after asking some questions. The following was tested on a…

15
r/LocalLLaMA community 4d ago

[Megathread] Qwen3.8-Flash-Next - Release Day

Megathread for discussing the (impending) release of Qwen 3.8 Flash Next. Quants Fine-Tunes & Abliterations Chat Templates Inference Server Support & Configuration Experiences, Benchmarks & Model Comparisons We'll try to clean up future duplicates around the release and point…

37
r/LocalLLaMA community 4d ago

First serious confirmation. Ox Alpha is GLM-5.3-Flash

https://x.com/romanchernin/status/2092488160680751437?s=20 - Multimodal (Vision) - 1M Tokens Context Window - DeepSWE ~63%   submitted by   /u/MrWidmoreHK [link]   [comments]

14
r/LocalLLaMA community 4d ago

Underrated Muse Glimmer

Benchmarked qwen3.8 xhigh, medium and muse glimmer. Xhigh effort mode with qwen3.8 took almost 30hrs. (And still failed on 16 cases because of the 32K output token limit) Medium effort mode and muse glimmer were 3-4 hours each. But I'm actually surprised by the muse glimmer…

31
r/LocalLLaMA community 4d ago

Open Source Kernel in Qwen3.6-35B-A3B for AMD MI350X: 78,498 output tok/s on 8 GPUs

So here's the thing, almost everyone use NVIDIA to run their LLMs, we also do the same, a lot of people we've met use like RTX PRO 6000 or even H100, B300 It seems like everyone eyes is looking at NVIDIA. However we do the math that the raw power alone on AMD GPU MI350X is…

9
r/LocalLLaMA community 4d ago

Thomson Reuters releases Thomson-1.0-Small. A law and tax focused model

  submitted by   /u/RedditUsr2 [link]   [comments]

23
r/LocalLLaMA community 4d ago

Getting Qwen3.8-27B with decent speed on my 4080 with 16Gb card

I saw that Q2 is actually very good and produce real good results: https://youtu.be/WNMnbba35VI?is=UNokHqdY4bA5kDgw and I also saw how dflash2 make its running at generating >60 t/s with a 120k context lenght. https://youtu.be/RBlRTUwJMI4?is=LCtTHgkaiWnGfLv9 And I like what its…

33
r/LocalLLaMA community 4d ago

Fully quantized NVFP4 Qwen3.8-27B with QUASAR QAD

We're releasing a fully quantized NVFP4 version of Qwen3.8-27B. The checkpoint was trained using quantization-aware distillation (QAD) with QUASAR, our new QAT algorithm. We used the original BF16 model as the teacher and distilled the quantized model for 2,446 steps. The…

6
r/LocalLLaMA community 4d ago

Qwen3.8-27B IQ3_XXS wrote a correct multilayer TMM on a 16 GB Quadro — after 100 minutes, 3 compactions, and 108k output tokens

https://preview.redd.it/i8rjx0ar5mlh1.png?width=2160&format=png&auto=webp&s=c2588bc7b2519ea71b176ca73faf566dfc585496 I wanted to see whether a heavily quantized 27B model running entirely on an older 16 GB workstation GPU could do more than the usual coding demos. FFT felt too…

8
r/LocalLLaMA community 4d ago

M5 Ultra 96GB vs M5 Max 128GB — is 2x bandwidth worth losing 32GB of RAM, with Qwen3.8-Flash-Next dropping tomorrow?

I’ve been going back and forth on this for a week and I can’t settle it, so I’m hoping someone here has hands-on numbers. The two configs (German prices, dealer quote, incl. VAT): Config Price Mac Studio M5 Max, 128GB / 512GB SSD €5,859 Mac Studio M5 Max, 128GB / 1TB SSD €6,189…

32
r/LocalLLaMA community 4d ago

Qwen 3.8 27b has ThreeJs locked down.

Generated on a 3090 Qwen 3.8 27b Q4 Thinking high.   submitted by   /u/Both_Opportunity5327 [link]   [comments]

19
r/LocalLLaMA community 4d ago

Peak Portable Personal Datacenter

Portable rig for Qwen3.8-27B-BF16 200K+ token prompts. My work Panasonic Toughbook + the T1 + power brick + headphones all fit in my lunchbox. Need the BF16 for huge context highly sensitive document OCR, image analysis, aggregation and summarization. I've done a ton of testing…

35
r/LocalLLaMA community 4d ago

It's here!

mrburns_excellent.gif   submitted by   /u/funding__secured [link]   [comments]

14
r/LocalLLaMA community 4d ago

35B-A3B tool calling benchmark: Original Qwen vs. KAT Coder, Ornith and Tiel-Coder

With hopes of a Qwen3.8-35B-A3B release now mostly dashed, many people including myself are looking at fine-tunes and other variants of Qwen3.6-35B-A3B to run on VRAM-limited hardware. I decided to try to benchmark some of the top contenders: KAT-Coder, Ornith 1.5 and the very…

35
r/LocalLLaMA community 4d ago

How to run LLMs as regular guy with low resources?

Hello everyone. I hope is all well with you all. I been around this sub for a few months and been quietly reading and I have seen how many of you are spending $10k on the Mac M5 and I am assuming you are using it for work or just for fun but have the resources to spend like…

11
r/LocalLLaMA community 4d ago

Granite Speech 5.0 Turbo CTC: Extremely Fast and Accurate Transcription

  submitted by   /u/coder543 [link]   [comments]

4
r/LocalLLaMA community 4d ago

You think they could have tweaked the typeface a bit?

  submitted by   /u/indicava [link]   [comments]

18
r/LocalLLaMA community 4d ago

Qwen3.8-Flash-Next. This architecture could be surprisingly local-friendly once the weights drop. 👀

Qwen3.8-Flash-Next (~125B-A6B + 51B n-gram) memory estimate: Ideal 4-bit quant ≈ 82 GB (58 GB main weights + 24 GB n-gram tables) Real-world quants likely land in the 80–90 GB range. The big n-gram table is sparsely accessed → excellent candidate for system RAM offload. This…

36
r/LocalLLaMA community 4d ago

CNBC Television: Nvidia partner with Perplexity AI to run locally in DGX Spark.

  submitted by   /u/dd32x [link]   [comments]

8
r/LocalLLaMA community 4d ago

Mac Studio M5 Max Cost Analysis

At $10k, you could get - 6.2B tokens with Qwen 3.8 Max (Qwen Pro plan) - 5.7B tokens with DeepSeek V4 Pro OpenRouter - 100B tokens with DeepSeek V4 Flash OpenRouter As a firm believer of local inference, unless you need it for data sovereignty, it's much more cost effect to wait…

26
r/LocalLLaMA community 4d ago

NInfer 4090 Windows update is out with 1.5-2k t/s prefill, extended MTP, disk caching with DirectStorage, built-in llama.cpp WebUI and more

I've made a few changes here and there to get nearly 2.1k tokens/sec prefill, ~210-230 tokens/sec decode with MTP7 (configurable, extended up to 15) on benchmarks. Also added disk caching options, up to 30GB per config by default for near-instant loads after server restart,…

10
r/LocalLLaMA community 4d ago

ibm-granite/granite-4.2-30b · Hugging Face

Granite-4.2-30B is the flagship reasoning model in the Granite 4.2 family. It delivers the strongest performance across reasoning-intensive tasks by leveraging built-in <think>...</think> chain-of-thought. It supports flexible thinking modes — full thinking (default),…

16
r/LocalLLaMA community 4d ago

12 abliterated Gemma 4 12B variants, one base, 165 GPU hours - Abliterlitics

I ran 11 uncensored variants of Gemma 4 12B that I grabbed from huggingface, sorting by downloads. 10 full abliterations plus 2 LoRA adapters which were requested to be added in the comparison, against the official base. 165 GPU hours over three and a half weeks on a single…

29

GLM-5.3 weights will be released tomorrow

this is a friendly reminder you can legally seed ai models via torrenting.

N-gram vs Experts explained

Local LLM harness for reverse engineering software?

Nvidia has been in talks to acquire Hugging Face for more than $13 billion - Business Insider

Qwen 27b ud IQ3XXS potential- 3D Zen Room demo - pi harness - build deploy and share link on discord

Qwen3.8 27B C8 at 972 TG / 5,680 PP on 4x MI100 rig ($6.5k) using my new INT8 vLLM fork

How to Fine-Tune an LLM: An End-to-End Guide

Any news about DeepSeek V4 Flash Vision weights?

Compared Qwen 3.8 27B community quants on RTX 6000 vs Claude Opus 4.6

Lemonade end-of-summer project update, now serving 15 engines!

Anyone else doing eGPUs (OCuLink)?

Self-hosting LLMs on budget hardware: general principles, hardware, benchmarks and frontends

Can we reconsider the megathreads?

Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

Are models with N-Gram tables going to completely change the AI race?

little tool for offline wikipedia RAG

Whoever the fuck predicted we would have gpt 5.5 performance in coding on consumer hardware a couple months ago now, i applaud you

[Megathread] GLM-5.3-Flash - former ox-alpha

What's your most reliable model, even if it's "outdated"?

Gemma4 31B vs Qwen3.8 27B - why the huge difference in benchmarks?

HF exploring sale - impact on open models?

zai-org/GLM-5.3-Flash · Hugging Face

GLM-5.3-Flash: Frontier Intelligence, Flash Cost

A minecraft clone I fully vibecoded with Qwen3.8-27b Q4

Forget the Pelican, it's Weevil-Time! / Benchmaxxing-Proof SVG and Vision Benchmark

A 27b model beating latest frontier models was not on my 2026 bingo card

OpenCode with Qwen3.8-27B for Small Games or Browsing the Web With 16GB VRAM

[Megathread] Qwen3.8-Flash-Next - Release Day

First serious confirmation. Ox Alpha is GLM-5.3-Flash

Underrated Muse Glimmer

Open Source Kernel in Qwen3.6-35B-A3B for AMD MI350X: 78,498 output tok/s on 8 GPUs

Thomson Reuters releases Thomson-1.0-Small. A law and tax focused model

Getting Qwen3.8-27B with decent speed on my 4080 with 16Gb card

Fully quantized NVFP4 Qwen3.8-27B with QUASAR QAD

Qwen3.8-27B IQ3_XXS wrote a correct multilayer TMM on a 16 GB Quadro — after 100 minutes, 3 compactions, and 108k output tokens

M5 Ultra 96GB vs M5 Max 128GB — is 2x bandwidth worth losing 32GB of RAM, with Qwen3.8-Flash-Next dropping tomorrow?

Qwen 3.8 27b has ThreeJs locked down.

Peak Portable Personal Datacenter

It's here!

35B-A3B tool calling benchmark: Original Qwen vs. KAT Coder, Ornith and Tiel-Coder

How to run LLMs as regular guy with low resources?

Granite Speech 5.0 Turbo CTC: Extremely Fast and Accurate Transcription

You think they could have tweaked the typeface a bit?

Qwen3.8-Flash-Next. This architecture could be surprisingly local-friendly once the weights drop. 👀

CNBC Television: Nvidia partner with Perplexity AI to run locally in DGX Spark.

Mac Studio M5 Max Cost Analysis

NInfer 4090 Windows update is out with 1.5-2k t/s prefill, extended MTP, disk caching with DirectStorage, built-in llama.cpp WebUI and more

ibm-granite/granite-4.2-30b · Hugging Face

12 abliterated Gemma 4 12B variants, one base, 165 GPU hours - Abliterlitics