r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 14d ago

Index-Translate: 150 text languages, plus document translation, multilingual subtitles and dubbing

I'm part of the BiliBili Index LLM team . We're sharing Index-Translate and its companion models for translating text, documents, and videos. Index-Translate supports 150 text languages , with 2B, 9B, and 35B-A3B (preview) options. You can specify terminology, writing style, and…

37
r/LocalLLaMA community 14d ago

Qwen3.8-27B-pi: Effort-Ordered Reasoning for Agentic Coding

  submitted by   /u/paf1138 [link]   [comments]

5
r/LocalLLaMA community 14d ago

Update: Strix Halo + R9700 with llama-halo-hybrid - now beats DGX Spark

Hi folks, I've spent the last couple of months experimenting with Strix Halo and previously I released a proof of concept I called llama-halo-hybrid. I've continued updating it and it now performs very well. The idea is that you can take an R9700, or similar, and place dense…

22
r/LocalLLaMA community 14d ago

Preorder for new AMD Ryzen™ AI Max 400 Series 192GB from framework just started

Framework Desktop Framework Desktop DIY Edition (AMD Ryzen™ AI Max 400 Series) 192GB   submitted by   /u/Educational_Sun_8813 [link]   [comments]

18
r/LocalLLaMA community 14d ago

What's going on with DGX Spark? Price up $2k in 1 week?

I need to buy 2x DGX Spark but can't find a seller. Any hope or idea where I can get 2? The price seems to have soared. Local Microcenter had 25+ then 0 the next day. They pulling stock?   submitted by   /u/StartupTim [link]   [comments]

34
r/LocalLLaMA community 14d ago

You can now run Qwen 3.8 27B on AMD NPUs via FastFlowLM (at a killer 1 tps decode)

  submitted by   /u/TuskNaPrezydenta2020 [link]   [comments]

14
r/LocalLLaMA community 14d ago

Another Ling model comes out, same receipt, 2 weeks free to use, then open source. Chinese labs do contribute a lot to open source community

Ling-3.1-flash: ~560B total params, ~25B active/token, up to 1M-token context. Across work, coding & healthcare: 1,673 Elo on GDPVal-AA v2.1, 75.16 on FrontierSWE, and 65.35 on HealthBench Professional.   submitted by   /u/Elouakili_Flexy [link]   [comments]

16
r/LocalLLaMA community 14d ago

Ling-3.1-flash: ~560B total params, ~25B active/token, up to 1M-token context- Free trail for 2 weeks then open source

  submitted by   /u/Muhlwa_Sholanke [link]   [comments]

11
r/LocalLLaMA community 14d ago

What was the mafia meeting of the tech criminals about at the White House?

Or should I assume it was mainly about trying to stop/ban/regulate China and open models,   submitted by   /u/XiRw [link]   [comments]

23
r/LocalLLaMA community 14d ago

How smart is the IQ3 family of Qwen 3.8 Flash Next for coding tasks?

I've been closely following the rise of the Strata inference engine and as someone with 28GB VRAM and 32GB RAM I'm itching to buy 32GB more RAM just to use Flash Next. But of course before I make such a financial commitment as a member of the GPU-poor class like myself, first I…

14
r/LocalLLaMA community 14d ago

Gliner2.5-Decide (Jev style model)

I was perusing huggingface trending and was surprised that this hadn't been posted to localllama. Looks like it's been out for about a week.   submitted by   /u/parepeg [link]   [comments]

17
r/LocalLLaMA community 14d ago

i would like to learn deeply about fine-tuning local models before burning money

There's so many new techniques like RL, RL LoRA, QLoRA, CPT LoRA. I believe i would have a usecase for them, but i don't know where to learn, youtube is filled with bad quality tutorials if i just search and the good channels (fireship, bycloud) don't cover these as they're…

24
r/LocalLLaMA community 14d ago

Update: Yandex/AliceAI 80B-A3B fine tune progress

loss curve (taken from the last micro of every step, to explain the variation) some help from gemini 3.8 flash high About 40% of the way done with the initial fine tune. The loss is so spiky because I accidentally used the last loss of each micro, rather than the average of each…

24
r/LocalLLaMA community 14d ago

We just open-sourced the world's fastest WebGPU kernels for local AI on Hugging Face

The collection includes kernels for more than 200 common ML operations, all of which can run entirely locally in your browser on WebGPU. We're also working to upstream these optimizations to Transformers.js, ONNX Runtime Web, LiteRT.js, and more! Kernels:…

14
r/LocalLLaMA community 14d ago

Follow-up: my native Rust + Vulkan Transformer training backend — 14 days later, now 14 parity-verified architectures and full PEFT

Follow-up to my post from about two weeks ago. A lot has changed since then, so I wanted to post an update on where the backend is now. Where the green architectures stand When I posted last time, 7 architectures had verified full training support. Everything is now held to the…

9
r/LocalLLaMA community 14d ago

add GLM-5.3-Flash (GLM5-Next) support (#27773) · ggml-org/llama.cpp@649dcb1

Finally!   submitted by   /u/challis88ocarina [link]   [comments]

26
r/LocalLLaMA community 14d ago

Oído: speech recognition that beats Whisper-tiny, running on a $5 microcontroller (open source)

I'm part of the Lokutor team that built this. Model: NVIDIA Conformer-CTC Small (13M params, int8). It runs on an ESP32-S3 with 8 MB PSRAM, no GPU or NPU. LibriSpeech WER is 3.7 / 8.2, versus 6.3 / 15.9 for Whisper tiny.en on a laptop. Under real noise (DEMAND: car, kitchen,…

16
r/LocalLLaMA community 14d ago

add GLM-5.3-Flash (GLM5-Next) support by timkhronos · Pull Request #27773 · ggml-org/llama.cpp

now you can use GLM-5.3-Flash on your home computer   submitted by   /u/jacek2023 [link]   [comments]

19
r/LocalLLaMA community 14d ago

AA-AgentPerf-Local: Benchmarking local AI agents on laptops and workstations

  submitted by   /u/nuclearbananana [link]   [comments]

16
r/LocalLLaMA community 14d ago

BAAI/AREX-2 - 27B - Agent model based on Qwen3.8 27B

"AREX-2 is a 27B-parameter long-horizon agent model from the Beijing Academy of Artificial Intelligence (BAAI). It learns to improve a solution over multiple test-time rounds: propose, measure, reflect, and revise. AREX-2 is trained on machine-learning and…

14
r/LocalLLaMA community 14d ago

DeepSeek now trained on Ascend 950

26 months ago Liang Wenfeng said: "Someone must step onto the frontier." Now they are training their models on Ascend 950   submitted by   /u/WebAssemblyMan [link]   [comments]

10
r/LocalLLaMA community 14d ago

1248 GB/s on 5060ti (+40%) with +5500 memory overclocks

There now is unlock for higher memory overclocks called mlock. And apparently the GDDR7 has a lot of headroom: https://www.reddit.com/r/overclocking/comments/1wsnllh/finally_unlocked_gddr7_memory_overclocking_with/ Of course this can help massively for local inference,…

34
r/LocalLLaMA community 14d ago

Thank You, Mradermacher.

best iq quants in the biz, got me gemma 4 26b to run 75tok/s tg and 1500 pp on 2x 4060 8gb using lmstudio serving to hermes, much work has been done.   submitted by   /u/Spiritual_Impress_30 [link]   [comments]

24
r/LocalLLaMA community 15d ago

Qwen3.8 flash next ISTA-DASLab GGUF 50t/s TG and 1500t/s PP with 12GB VRAM and 64GB RAM Laptop on 'Strata' engine

I think most people are sleeping on this inference engine. I tried multiple llama.cpp forks and none of them comes close to the inference speed of Strata. Initial version had some bugs with kv cache, cpu throttling and the developer fixed them. Inference engine (only runs on…

15
r/LocalLLaMA community 15d ago

Best approach for automatically tagging local music collection?

I don't use music streaming services much, and listen to music from my own local collection. I don't use any local streaming servers like Plex or Navidrome, they wouldn't work for me because I use a dumbphone and play music off of my SD card. I've manually built a bunch of mood…

16
r/LocalLLaMA community 15d ago

PSA: ModelScope CLI is now moved to "modelscope-hub"

To save people 30 minutes of research (because they didn't bother documenting this officially at all): The "modelscope" package is now just the library. Doesn't contain a CLI anymore. If you try to install it or update your old CLI package, you get "No executables are provided…

5
r/LocalLLaMA community 15d ago

SFTMill: Easily [off-policy] distill any existing LLM with an OpenAI Compatible Endpoint. Turn any behavioral goal into a comprehensive dataset.

Disclaimer: Any* means any model that exposes its CoT without it being censored. Hey guys - half a tutorial/guide, and half an I built this, so I went for resources. This is something that I created for myself recently when I couldn't find any good existing solution. I wrote…

21
r/LocalLLaMA community 15d ago

Qwen-family LLMs are quietly becoming the backbone of modern audio models; One chart for the architectures of 100+ audio models

I started mapping the building blocks shared across all the models in audio.cpp. The result ended up being more interesting than I expected. Qwen has become by far the most common language backbone in this collection: 32 audio model families use a Qwen-family architecture, and…

22
r/LocalLLaMA community 15d ago

Are you worried about a potential ban of Chinese open weight models?

Anthropic released the GLM article today. Trump is getting very involved. Do you foresee Chinese open weight models getting banned soon?   submitted by   /u/writesfw [link]   [comments]

27
r/LocalLLaMA community 15d ago

Sherry's 3:4 ternary format (1.375 bits per weight) running on WebGPU: a 1.6 MB model that plays Connect Four as well as its 7.8 MB int8 version

Not an LLM, but the ternary findings should carry over, and we hadn't seen Sherry-style 3:4 weights run in a browser before. Disclosure: this is our work at Precisit, everything is MIT. What it is A 7.4M-parameter one-pass scorer (the jevlike family): the board goes in, one…

19
r/LocalLLaMA community 15d ago

AMD boosting AI/LLM performance for Radeon iGPUs as much as 18~23% with Linux 7.4

  submitted by   /u/Fcking_Chuck [link]   [comments]

33
r/LocalLLaMA community 15d ago

exl3 now in ninfer-ext

My ninfer-ext fork of the famous ninfer now supports exl3 , and have released two models as well. https://huggingface.co/jabbatheduck/ninfer-ext-models   submitted by   /u/giveen [link]   [comments]

20
r/LocalLLaMA community 15d ago

Best model for blender?

Trying to see if I can use a local model to generate game assets, or even 3D printer models. Any suggestions? Something that would fit in 64GB of ram, speed is not an issue. Just need it to work well.   submitted by   /u/rorowhat [link]   [comments]

15
r/LocalLLaMA community 15d ago

Another case of censorship

I was optimizing my diet using a cloud AI provider, and I found my Pi agent stuck like this. Looks like I had too many mushrooms lol.   submitted by   /u/autoencoder [link]   [comments]

8
r/LocalLLaMA community 15d ago

What kinds of problems are still fundamentally hard for LLMs?

I played a game of a custom chess variant against an claude opus 5.5, and it beat me. The game combined several rule changes: the board wraps around from the h-file to the a-file, knights move three squares in one direction and one sideways, and captured pieces can be dropped…

31
r/LocalLLaMA community 15d ago

Anthropic just dropped the greatest advertisement for GLM ever.

Like.. yea bro, I knew GLM was cool. Now everyone does.   submitted by   /u/BannedGoNext [link]   [comments]

25
r/LocalLLaMA community 15d ago

Recommended replacements for glm 4.7 flash

I know I sound crazy, but i’m using a strix halo and finding that GLM 4.7 flash just runs significantly better than qwen 3.6 35 a3b. But its obviously quite old at this point, i wish we had a new glm that was 4.7 flash sized but is anyone using a model that they have found…

21
r/LocalLLaMA community 15d ago

Inference Engines will become a series of one-offs

ninfer, dwarfstar, Splash, llamAmpere, gufo, etc. We've all seen them popping up, great tok/s, people loving them. Forks of llama.cpp or another engine, or made from scratch. For better or worse, the list will continue to grow They work so well because they dodge a main…

22
r/LocalLLaMA community 15d ago

Deepseek Harness app is out now!!!

Downloading now.   submitted by   /u/politefella0 [link]   [comments]

9
r/LocalLLaMA community 15d ago

RAM Offloading with vLLM - tcclaviger appreciation post

Thanks to tcclaviger, vLLM now has expert RAM offloading support ( link ). This makes frontier models much more accessible on a local setup! I was able to run the original DeepSeek-V4-Flash-Vision-Exp on four R9700s. podman run --rm -it \ --init \ --network host \ --ulimit…

27
r/LocalLLaMA community 15d ago

GLM-5.3 and the Spread of Advanced Cyber Capabilities \ Anthropic

  submitted by   /u/brown2green [link]   [comments]

23
r/LocalLLaMA community 15d ago

Thanks to you r/LocalLLaMA, my mom was able to use my app! The open-source app that can watch your screen and trigger actions. It is now easy to use, thanks to your feedback.

TL;DR: I'm a solo dev who wanted a simple, private way to have local LLMs watch my screen and do simple logging/notifying. After a year of building, I released v3.0.0 and my mom was able to use it for the first time and I wanted to say thank you! Hey r/LocalLLaMA , What is it…

9
r/LocalLLaMA community 15d ago

AMD's new 256 core EPYC has 16-channel DDR5-12800, 91% memory bandwidth of an RTX 5090

Here are some inspirational quotes you can put into the comments: God is dead and we killed him I am become death All this for 1.5 tok/s? Sir this is LocalLLaMA not RichPeopleofLocalLLaMA Sweet! A 2TB DDR5-12800 RDIMM kit is going to cost only 2 kidneys and a small micronation's…

35
r/LocalLLaMA community 15d ago

Help me find a good stack for reversing an old online game client

Hi, so I was working on building a local server for an older online game client a few years ago, and the amount of data I had to synthesize was intense. I ended up shelving the project. I had managed to build a basic login server, sorted out some crypt stuff, but it was just way…

5
r/LocalLLaMA community 15d ago

What are your experiences with using a hybrid cloud/local setup to stretch usage for coding projects?

For example, directly using claude code or code, which is then hooked up to automatically delegate the actual code writing tasks to a local model like qwen 3.8 flash next, to save on cloud usage limits. I’m imagining the loop would be: User writes prompt Claude/codex thinks…

18
r/LocalLLaMA community 15d ago

Reflection 70B was released two years ago (September 2024)

You may think that jev, OpenClaw or TurboQuant are super cool , but actually the coolest LLM invention happened two years ago As we all know, the best source of reliable information about LLMs is YouTube:…

18
r/LocalLLaMA community 15d ago

Qwen 3.8 27B Q4 with 100K context on a 16 GB RX 7800 XT guide

I'm running Qwen 3.8 27B Q4 XS with ~30 t/s decode (no MTP) at 100k context with q8/q5 KV cache on a 16GB AMD GPU. I wanted to share my setup since many believe Qwen 3.8 27B to be infeasible on 16GB VRAM. Build llama.cpp with Vulkan: cmake -B build -DGGML_VULKAN=ON && cmake…

4
r/LocalLLaMA community 15d ago

IQuestLab/IQuest-Q1 · Hugging Face

IQuest-Q1 is a Mixture-of-Experts (MoE) model developed by IQuest for agentic coding, reasoning, and multi-step tool use. It comprises approximately 320B total parameters , with an estimated 15B parameters activated per token .   submitted by   /u/jacek2023 [link]  …

38
r/LocalLLaMA community 15d ago

50B+ MoEs with few active parameters, what's the sweet spot for intelligence, agent speed, and affordable fine-tuning?

I’m building a Polish General purpose legal Model that drafts documents, answers questions using legal sources, and has enough coding ability to handle some automation. The workflow is very tool-heavy: Question → many sequential tool calls → final answer/document Think Claude…

5
r/LocalLLaMA community 15d ago

Looks like the era of subsidised compute is coming to an end. The old ChatGPT Pro $200 20x plan will be halved. The new $500 plan will have similar limits as the (old) $200 plan.

  submitted by   /u/Norwood_Reaper_ [link]   [comments]

36

Index-Translate: 150 text languages, plus document translation, multilingual subtitles and dubbing

Qwen3.8-27B-pi: Effort-Ordered Reasoning for Agentic Coding

Update: Strix Halo + R9700 with llama-halo-hybrid - now beats DGX Spark

Preorder for new AMD Ryzen™ AI Max 400 Series 192GB from framework just started

What's going on with DGX Spark? Price up $2k in 1 week?

You can now run Qwen 3.8 27B on AMD NPUs via FastFlowLM (at a killer 1 tps decode)

Another Ling model comes out, same receipt, 2 weeks free to use, then open source. Chinese labs do contribute a lot to open source community

Ling-3.1-flash: ~560B total params, ~25B active/token, up to 1M-token context- Free trail for 2 weeks then open source

What was the mafia meeting of the tech criminals about at the White House?

How smart is the IQ3 family of Qwen 3.8 Flash Next for coding tasks?

Gliner2.5-Decide (Jev style model)

i would like to learn deeply about fine-tuning local models before burning money

Update: Yandex/AliceAI 80B-A3B fine tune progress

We just open-sourced the world's fastest WebGPU kernels for local AI on Hugging Face

Follow-up: my native Rust + Vulkan Transformer training backend — 14 days later, now 14 parity-verified architectures and full PEFT

add GLM-5.3-Flash (GLM5-Next) support (#27773) · ggml-org/llama.cpp@649dcb1

Oído: speech recognition that beats Whisper-tiny, running on a $5 microcontroller (open source)

add GLM-5.3-Flash (GLM5-Next) support by timkhronos · Pull Request #27773 · ggml-org/llama.cpp

AA-AgentPerf-Local: Benchmarking local AI agents on laptops and workstations

BAAI/AREX-2 - 27B - Agent model based on Qwen3.8 27B

DeepSeek now trained on Ascend 950

1248 GB/s on 5060ti (+40%) with +5500 memory overclocks

Thank You, Mradermacher.

Qwen3.8 flash next ISTA-DASLab GGUF 50t/s TG and 1500t/s PP with 12GB VRAM and 64GB RAM Laptop on 'Strata' engine

Best approach for automatically tagging local music collection?

PSA: ModelScope CLI is now moved to "modelscope-hub"

SFTMill: Easily [off-policy] distill any existing LLM with an OpenAI Compatible Endpoint. Turn any behavioral goal into a comprehensive dataset.

Qwen-family LLMs are quietly becoming the backbone of modern audio models; One chart for the architectures of 100+ audio models

Are you worried about a potential ban of Chinese open weight models?

Sherry's 3:4 ternary format (1.375 bits per weight) running on WebGPU: a 1.6 MB model that plays Connect Four as well as its 7.8 MB int8 version

AMD boosting AI/LLM performance for Radeon iGPUs as much as 18~23% with Linux 7.4

exl3 now in ninfer-ext

Best model for blender?

Another case of censorship

What kinds of problems are still fundamentally hard for LLMs?

Anthropic just dropped the greatest advertisement for GLM ever.

Recommended replacements for glm 4.7 flash

Inference Engines will become a series of one-offs

Deepseek Harness app is out now!!!

RAM Offloading with vLLM - tcclaviger appreciation post

GLM-5.3 and the Spread of Advanced Cyber Capabilities \ Anthropic

Thanks to you r/LocalLLaMA, my mom was able to use my app! The open-source app that can watch your screen and trigger actions. It is now easy to use, thanks to your feedback.

AMD's new 256 core EPYC has 16-channel DDR5-12800, 91% memory bandwidth of an RTX 5090

Help me find a good stack for reversing an old online game client

What are your experiences with using a hybrid cloud/local setup to stretch usage for coding projects?

Reflection 70B was released two years ago (September 2024)

Qwen 3.8 27B Q4 with 100K context on a 16 GB RX 7800 XT guide

IQuestLab/IQuest-Q1 · Hugging Face

50B+ MoEs with few active parameters, what's the sweet spot for intelligence, agent speed, and affordable fine-tuning?

Looks like the era of subsidised compute is coming to an end. The old ChatGPT Pro $200 20x plan will be halved. The new $500 plan will have similar limits as the (old) $200 plan.