r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 4d ago

Intel Arc Pro B60 Dual 48G spotted

I spotted the Dual B60 48GB listed on Digitec/Galaxus. Initially it was said these wouldn't go into standard retail channels. At CHF 2500 (post tax, USD ~3000) not particularly competitive but worth keeping an eye on. For it to be interesting it shouldn't be more than like 2.5x…

25
r/LocalLLaMA community 4d ago

me to the model I spent all weekend fine-tuning

I just can't resist   submitted by   /u/close_Meal6005 [link]   [comments]

27
r/LocalLLaMA community 4d ago

Apple unveils a more powerful Mac mini featuring the all-new M6 and M5 Pro

"A 12-core GPU, also with two more cores than before, now includes Neural Accelerators in each core for the first time on Mac mini, resulting in up to 4x faster AI performance and 2x faster graphics than Mac mini with M4. In addition, the all-new Dual 16-core Neural Engine…

31
r/LocalLLaMA community 5d ago

Qwen 4 architecture: What do we know?

My best bet is the embedding-offloaded linear where the 51b n-grams track semantics and context, like 3.8 with its loss of real world knowledge bolted back on. Otherwise? Sparse full-attention with dense routing? 6b active route to 'heavy' layers when the n-gram gets stuck.…

5
r/LocalLLaMA community 5d ago

Apple releases M5 ultra at 1.2TB/s bandwith

lpddr5x probably, the m7 ultra if is using ddr6 should be at 1.8 Tb/s   submitted by   /u/Last-Owl-8342 [link]   [comments]

4
r/LocalLLaMA community 5d ago

Apple introduces new Mac Studio with M5 Max and M5 Ultra - up to 512GB of unified memory

  submitted by   /u/themixtergames [link]   [comments]

18
r/LocalLLaMA community 5d ago

Quantization-Aware Healing: a compressed, 4-bit model that outperforms its full-precision original

  submitted by   /u/Decent-Hat-5807 [link]   [comments]

26
r/LocalLLaMA community 5d ago

Qwen 3.8 Flash Next day 0 support from unsloth

Prepare your disk space guys   submitted by   /u/jacek2023 [link]   [comments]

18
r/LocalLLaMA community 5d ago

New: Llama.cpp adaptive speculation for faster inference

We have been working on some performance optimisations for Qwen3.8 and other models. The main new feature that we introduced is adaptive speculation for Llama.cpp What is it? MTP and DFlash work well to speed up inference work, especially for dense models. However, different…

36
r/LocalLLaMA community 5d ago

Qwen3.8 flash next

  submitted by   /u/RuthlessCriticismAll [link]   [comments]

25
r/LocalLLaMA community 5d ago

Qwen3.8-Flash-Next tomorrow

  submitted by   /u/rerri [link]   [comments]

15
r/LocalLLaMA community 5d ago

Glm 5.3 flash?

glm 5.3 flash While awaiting the release of the version 5.3 weights, this theory is gaining ground. OxAlpha is new GLM.   submitted by   /u/LegacyRemaster [link]   [comments]

13
r/LocalLLaMA community 5d ago

tencent/WeMM-Embedding 9B/4B/2B

WeMM-Embedding-9B is a universal multimodal embedding model built on Qwen3.5. It accepts text, images, videos, visual documents, and interleaved multimodal inputs, and returns a 4,096-dimensional L2-normalized embedding. Audio input is not supported.…

10
r/LocalLLaMA community 5d ago

How many of you are using Kiwix/offline Wikipedia for training?

We are seeing every so often people mentioning using Kiwix or offline Wikipedia for training. It is not something Kiwix is actively involved in (as in, people found us rather than the other way around, and no one on the team is working on offline LLMs except as a side hobby),…

38
r/LocalLLaMA community 5d ago

Scaffold CoT: A CoT dataset built around the failures of small model (>5B Params) free form thinking. Hope its useful to you guys!

TL;DR - A ~4M example, ~3B token CoT dataset designed around helping small models think more concisely, accurately and reliably. Hi all! For the past few months I have been working on a dataset designed around improving small model performance through a structured framework (or…

5
r/LocalLLaMA community 5d ago

To all of you who have bought Chinese ASICs, how have they been?

as we all know to get anything good and modern for nvidia/amd if you are lucky u can give up your kidneys as a down payment, but the chinese accelerators have a huge value proposition, if ur willing to invest the time and tokens porting frameworks to them. To whoever owns them,…

15
r/LocalLLaMA community 5d ago

[2608.16157] FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution

Source of Claims: https://x.com/Andy_ShuoYang/status/2090856976880472439 Your gaming PC can now serve frontier models at interactive speed using official checkpoints without extreme quantization! Qwen3.6 35B → 8GB RTX 4060 laptop @ 39 tok/s DeepSeek-V4-Flash 284B → RTX 5090…

22
r/LocalLLaMA community 5d ago

I just tried DeepSeek Harness and it escaped from its workspace folder

It worked pretty well, digging through and analyzing some local files. Claude code regularly stops at some point and fails to continue while DSH worked for 2 h, recognized that it could benefit from reading more context and ... bummer: It left the project folder (although DSH…

34
r/LocalLLaMA community 5d ago

The journey of letting Qwen 3.6/3.8 autonomously coding a c compiler.

Hi, Back in late march I begun playing around with Qwen 3.6 27b and found like everyone else that it's notoriously good at tool calls, where every model I tried before just derailed after a few turns it kept going and felt quite reliable outside of typical behaviors of smaller…

22
r/LocalLLaMA community 5d ago

Qwen-3.8-27B, Nemotron-3.5-Lightning-30B-A3B, Ornith-1.5-35B-A3B, Muse-Glimmer-30B oQ8e comparison

Ornith does really well. TielCoder ( https://llm-bench.io/benchmarks/cmt7kp2zj002r01lcmpchvlko ) might be even a bit better in coding. Will give it a try soon. Details of the comparison see here:…

35
r/LocalLLaMA community 5d ago

Copilot you say?

talking to any white collar employee   submitted by   /u/edge_compute_user [link]   [comments]

10
r/LocalLLaMA community 5d ago

JetBrains local AI (using Qwen3.6 27B)

Sounds quite interesting, a big IDE provider optimizing for local AI with their coding harness. Especially that they picked Qwen3.6 over Qwen3.8 because of the thinking needs. Haven't read the full article yet, but sounds really cool.   submitted by   /u/Danmoreng [link]…

36
r/LocalLLaMA community 5d ago

Please join r/LowEndLocalAI, a community for running local LLMs on low spec hardware

If you’re trying to run local LLMs on a normal laptop, an older desktop, integrated graphics, limited VRAM, or simply the hardware you already own, r/LowEndLocalAI is meant for you. The idea is simple: What useful things can we do with the hardware we already have? I’ve been…

24
r/LocalLLaMA community 5d ago

Planning to spend ~$100 benchmarking differnet Qwen3.8-27B quants and kv cache and looking for input before I start

TL;DR: I'm planning to spend around $100 on cloud GPUs to benchmark Qwen3.8-27B with a focus on questions that actually matter when running it locally: different quant levels/providers, 8-bit vs 16-bit KV cache, GGUF vs EXL3, context length tradeoffs, and token efficiency on…

29
r/LocalLLaMA community 5d ago

MobileMoE - a facebook Collection

MobileMoE is a family of on-device Mixture-of-Experts (MoE) language models with sub-billion active parameters, designed to push the quality–efficiency Pareto frontier for on-device LLMs, including three model scales (S/M/L): 0.3B/0.5B/0.9B active parameters (1.3B/2.8B/5.3B…

7
r/LocalLLaMA community 5d ago

One LLM wrote a trading feature. Another reviewed it. Both missed a future-data bug

An AQuA research loop let one LLM write an intraday trading feature and a second LLM review it for causality. The feature divided the volume seen so far by the day's final total volume. At prediction time, that final total included trades from later in the day: future data. The…

24
r/LocalLLaMA community 5d ago

Do not blindly delete your older models, some are still precious

I have deleted tons and tons of older models to make space since I can't afford storage anymore. Easily 10TB... Anyways, I have been considering deleting DeepSeekV3.2 but decide to run it one more time. I have a problem I have been brainstorming about and have chatted locally…

27
r/LocalLLaMA community 5d ago

This is what Qwen 3.8 27b is capable of

Try it here: https://ocean.blackbeardlabs.dev/ Model: Qwen 3.8 27b Q8_X_KL Unsloth Hardware: 3 x RTX3090 Harness: DeepSeek Harness Prompt: /goal I want you to create a **JavaScript + Node.js WebGL project** that renders a highly realistic real-time ocean in the browser. Use…

14
r/LocalLLaMA community 5d ago

Qwen 3.8 27B in 9th position on code arena. Gemma 4 31B is 80th.

  submitted by   /u/tarruda [link]   [comments]

16
r/LocalLLaMA community 5d ago

Bart: A vintage llm

after 3 months and $800 burned... Unbounded Labs is proud to introduce Bart, our vintage LLM: 2.82B parameters trained from scratch on 20.1B tokens of English written before 1931. You can talk to it right now! Demo: https://www.unboundedlab.com/chat/bart Article:…

35
r/LocalLLaMA community 5d ago

Apple M5 Server

Credit to Twitter Post   submitted by   /u/Rymssss [link]   [comments]

9
r/LocalLLaMA community 5d ago

What's the best local model you've found for 8 GB of VRAM?

I'm curious what other people are using for local LLM coding / agentic coding with only 8 GB of VRAM . My current setup is: Intel Core i7-11800H RTX 3070 Laptop , 8 GB VRAM 32 GB DDR4 RAM openSUSE Tumbleweed / KDE Unsloth Studio pi.dev as the coding agent After testing quite a…

33
r/LocalLLaMA community 5d ago

llama.cpp docs now have a new home ❤️

  submitted by   /u/unofficialmerve [link]   [comments]

37
r/LocalLLaMA community 5d ago

[Paper] ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning

ToMoE : Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning Large Language Models (LLMs) have demonstrated remarkable abilities in tackling a wide range of complex tasks. However, their huge computational and memory costs raise…

19
r/LocalLLaMA community 5d ago

TielCoder's 22 GB 4-bit quant matches Opus4.6 medium on recent real life coding issues, surpassing KAT-Coder and Nail as strongest and fastest MoE picks.

Qwen3.8-27B is amazing, but it’s slow. A stronger 35B-A3B Mixture of Experts-coder that can run and solve real codebase issues fast (even on constrained hardware) is a valuable addition to the arsenal. This one is the strongest and most consistent 35B-A3B I’ve benchmarked, on…

38
r/LocalLLaMA community 5d ago

iPhone Local TTS EPUB Reading - Audiobookify

I've been working on this project (been a developer for a few years) for a few months now, and it has been in active testing for ~2 months. Its an EPUB reader that also offers local offline TTS, so its not just TTS focused, its meant to be a good regular reading app as well. I'd…

38
r/LocalLLaMA community 5d ago

Real local agentic coding on a 12GB VRAM budget.

Thanks to Unsloth Dynamic 3.0 quants coming in slightly leaner and better preserved, I settled on Qwen 3.8 27B (`UD_Q4_K_XL`) at 100K context as my daily driver for Hermes Agent and OpenCode. On an RTX 5070 Ti Mobile (12GB) paired with an Intel Core Ultra 9 275HX and 32GB DDR5,…

29
r/LocalLLaMA community 6d ago

I irradiated LLMs and found that they die really quickly

I randomly bit flipped a llm to simulate what would happen if you ran your spark in low earth orbit i hope it's ok to share this here, I was told this community might enjoy it.   submitted by   /u/BenniJesus [link]   [comments]

7
r/LocalLLaMA community 6d ago

Who would buy HuggingFace

Given OpenRouter.ai was snapped up by Stripe, who do we think would go after the "GitHib" of AI models? It is a big chunk of change they are looking ($13B). Apple may be a contender to give them a real chip in the AI race, given how they are focused on local AI execution.  …

11
r/LocalLLaMA community 6d ago

GPU Poor - Don't overlook Laguna XS 2.1

As per title. I'm not affiliated with the team behind this model in any way, shape or form. As a GPU poor myself (8 GB VRAM laptop + 12 GB VRAM desktop), I found Laguna to be very promising on my laptop. It runs at 30t/s (60k context) and it one-shot my two usual test prompts :…

19
r/LocalLLaMA community 6d ago

Top 10 most liked models on Qwen's HuggingFace page

This shows how Qwen3.8-27B smashed all expectations and in less than a month got more likes than the next 4 models combined, finally surpassing their long reigning queen QwQ-32B. Next best thing is Qwen3.6-35B-A3B (which they have been sleeping on during 3.8 iteration). The…

33
r/LocalLLaMA community 6d ago

At a certain point, speed >> smartness

It feels like a zig-zag: you don't want a model that's too dumb to do anything agentic. But once a model is good enough to be agentic, you don't want it to run so slow that iterating takes hours. For me the sweet spot is something like ~500 tps prefill, ~25tps decode. If a model…

24
r/LocalLLaMA community 6d ago

Qwen 3.8 27B Aider score

I ran the Aider benchmark on Qwen 3.8 27B FP8 with FP8 KV cache 256K context vLLM. The score: 72.9 This matches Gemini 2.5 Pro from 2025-04-12 which also scored 72.9. Beats Claude Opus 4 from 2025-05-25 which scored 72.0. DeepSeek R1 2025-06-06 scored 71.4. It may just be a…

24
r/LocalLLaMA community 6d ago

Xiaomi AI Cube announced with 1.2TB/s memory bandwidth

Xiaomi announced a prototype for their Xiaomi AI Cube. 3 chip system: - Xiaomi Xuanjie O3 - Xiaomi Xuanjie O100 - Xiaomi Xuanjie D100 The specs are impressive, but a bit confusing. The D100 chip (originally for their EVs) supports up to 160GB of RAM, but O100 has the 1.22TB/s…

20
r/LocalLLaMA community 6d ago

Hugging Face for sales? 👀

wonder what would happen to open models, if the sales went through, and hf got acquired by a big tech shop? 👀 https://www.businessinsider.com/hugging-face-could-be-acquired-13-billion-2026-8 $13 billion not a bad price though 😂   submitted by   /u/HugeConsideration211…

37
r/LocalLLaMA community 6d ago

Best AI Voice Cloning in 2026: How to Clone Your Voice With AI

  submitted by   /u/NextgenAITrading [link]   [comments]

14
r/LocalLLaMA community 6d ago

What are all the things qwen 3.8 27B is NOT good for?

What has it gotten worse at or what is it unusable for?   submitted by   /u/Adventurous-Gold6413 [link]   [comments]

21
r/LocalLLaMA community 6d ago

I developed my own quantized LLM from scratch, trained on 30B tokens, deploys in 60 MB

Reposting here after sharing this on [ r/MachineLearning ]( r/MachineLearning ) a few days ago, where it got a much better response than I expected (300+ upvotes, great questions, zero roasting) GitHub is at 35 stars now. So here it is. I trained a 250M parameter model from…

26
r/LocalLLaMA community 6d ago

deepseek-v4-flash-0731 - surprisingly usable

I just finished building my (relatively) low rent local inference machine: * Epyc 7663 * 256GB ECC DDR4-3200 * 1x RTX 5090 32GB Yeah I realize it's weird to throw a 5090 and 256GB of anything together and call it low end, but relative to ~151GB of weights it is. I'm running…

15
r/LocalLLaMA community 6d ago

I trained a 1.57B-parameter Dreamer 4 World Model from scratch for under $150

My first attempt didn't work. I built on Genie's architecture and the videos looked great, but the controls barely did anything. The effect of a keypress was basically zero. Genie learns its actions unsupervised into 8 codes, and that was too loose a grip for us. So I scrapped…

33

Intel Arc Pro B60 Dual 48G spotted

me to the model I spent all weekend fine-tuning

Apple unveils a more powerful Mac mini featuring the all-new M6 and M5 Pro

Qwen 4 architecture: What do we know?

Apple releases M5 ultra at 1.2TB/s bandwith

Apple introduces new Mac Studio with M5 Max and M5 Ultra - up to 512GB of unified memory

Quantization-Aware Healing: a compressed, 4-bit model that outperforms its full-precision original

Qwen 3.8 Flash Next day 0 support from unsloth

New: Llama.cpp adaptive speculation for faster inference

Qwen3.8 flash next

Qwen3.8-Flash-Next tomorrow

Glm 5.3 flash?

tencent/WeMM-Embedding 9B/4B/2B

How many of you are using Kiwix/offline Wikipedia for training?

Scaffold CoT: A CoT dataset built around the failures of small model (>5B Params) free form thinking. Hope its useful to you guys!

To all of you who have bought Chinese ASICs, how have they been?

[2608.16157] FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution

I just tried DeepSeek Harness and it escaped from its workspace folder

The journey of letting Qwen 3.6/3.8 autonomously coding a c compiler.

Qwen-3.8-27B, Nemotron-3.5-Lightning-30B-A3B, Ornith-1.5-35B-A3B, Muse-Glimmer-30B oQ8e comparison

Copilot you say?

JetBrains local AI (using Qwen3.6 27B)

Please join r/LowEndLocalAI, a community for running local LLMs on low spec hardware

Planning to spend ~$100 benchmarking differnet Qwen3.8-27B quants and kv cache and looking for input before I start

MobileMoE - a facebook Collection

One LLM wrote a trading feature. Another reviewed it. Both missed a future-data bug

Do not blindly delete your older models, some are still precious

This is what Qwen 3.8 27b is capable of

Qwen 3.8 27B in 9th position on code arena. Gemma 4 31B is 80th.

Bart: A vintage llm

Apple M5 Server

What's the best local model you've found for 8 GB of VRAM?

llama.cpp docs now have a new home ❤️

[Paper] ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning

TielCoder's 22 GB 4-bit quant matches Opus4.6 medium on recent real life coding issues, surpassing KAT-Coder and Nail as strongest and fastest MoE picks.

iPhone Local TTS EPUB Reading - Audiobookify

Real local agentic coding on a 12GB VRAM budget.

I irradiated LLMs and found that they die really quickly

Who would buy HuggingFace

GPU Poor - Don't overlook Laguna XS 2.1

Top 10 most liked models on Qwen's HuggingFace page

At a certain point, speed >> smartness

Qwen 3.8 27B Aider score

Xiaomi AI Cube announced with 1.2TB/s memory bandwidth

Hugging Face for sales? 👀

Best AI Voice Cloning in 2026: How to Clone Your Voice With AI

What are all the things qwen 3.8 27B is NOT good for?

I developed my own quantized LLM from scratch, trained on 30B tokens, deploys in 60 MB

deepseek-v4-flash-0731 - surprisingly usable

I trained a 1.57B-parameter Dreamer 4 World Model from scratch for under $150