r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 11d ago

I built a code knowledge graph tool that's actually MIT licensed (fully local, no cloud)

So this is maybe a niche problem, but at my job I work on a huge Python codebase and every time I change some shared function I'm basically playing roulette. grep tells who mentions it in the code base, not who actually calls it. And more essentially, Claude Code (my major…

15
r/LocalLLaMA community 11d ago

Sopro V2 Turbo 2610: cleaner cloned voices, same 120M model, same CPU speed

Follow-up to last month's post. One of the main issues people ran into was roughness or break-up on some cloned voices. 2610 is an interim update focused mostly on improving that. Reduced roughness and break-up on some of the voices that struggled before Same 120M model, same…

9
r/LocalLLaMA community 11d ago

Two local Qwen ( 3.8 27b unsloth Q6 and Qwen flash next strata coder ) models vs Claude Opus 4.6 on the same 3 coding tasks. One of them tied it. Not here to start a fight, just sharing numbers

Innanzitutto, due cose per evitare fraintendimenti. Non sto cercando di sostenere che un modello o un'azienda siano migliori. Non ho alcun interesse personale in nessuno di essi. Volevo solo verificare personalmente come si comportano nello stesso contesto lavorativo. E il…

19
r/LocalLLaMA community 11d ago

Come let your LLMs play World of Warcraft

I hosted my own world of warcraft private server then built a client that you can play in the browser on PC or mobile at https://jankcraft.xyz/ for free. Afterwards, I created a custom MCP and agent harness to control the browser client and play the game by sending signals over…

15
r/LocalLLaMA community 11d ago

POV 8GB VRAM + 16GB RAM folks checking locallama everyday:

One can only hope for Qwen4 35B A3B or similar (maybe with extra ngrams for 35B -> 70B+... until then, Gemma 26B QAT my beloved (solid 26 Tg/s)...   submitted by   /u/RobustLokiX [link]   [comments]

32
r/LocalLLaMA community 11d ago

Two ~300B MoE models, each on ONE 128 GB mini PC (AMD Strix Halo): GLM-5.3-Flash at ~580 tok/s prefill, MiMo-V2.6-Flash up to 44 tok/s decode. EXL3 weights + open ROCm engine

We built an engine, Kyojin, on top of ExLlamaV3 for Strix Halo (gfx1151, ROCm), and packed two 300B-class MoE models so each fits one 128 GB machine. First release, all measured on Ryzen AI Max+ 395. Model GLM-5.3-Flash MiMo-V2.6-Flash-MOPD Size 99.7 GB 105 GB Prefill 580 tok/s…

19
r/LocalLLaMA community 11d ago

Yes bots we get it, Strata is good now please stop

It's like the entire sub has become that scene from Konosuba where the cult keeps making up fake scenarios saying the only solution is to join their religion   submitted by   /u/Mayion [link]   [comments]

34
r/LocalLLaMA community 11d ago

Anyone using a local AI meeting notes setup instead of Fathom?

Meeting notes are one of the last parts of my workflow that still depend heavily on cloud tools. I've used Fathom and lately Bluedot. Bluedot works well for me because there's no meeting bot and I get the transcript, summary and action items after. But I'd really like to move…

27
r/LocalLLaMA community 11d ago

Qwen Flash next on 64GB RAM unified iGPU anyone ? (non-mac)

I've seen people reporting running it with 12GB VRAM + 64 GB RAM. Also with 64 GB RAM unified in Macs, but I was wondering if it's possible with any inference backend to run it for example on a 64 GB RAM minipc+ iGPU (780m in my case). I'm currently running 125B Ling 3.0 flash…

13
r/LocalLLaMA community 11d ago

Replacing vector databases with SQLite and SIMD hypervectors in under 1.2GB VRAM (Hillock)

Disclosure: I am the creator of this project. After days of lurking and building up enough karma, I can finally post here. Every time I tried running local RAG on my own machine, I hit the exact same bottlenecks. First, spinning up Chroma or another vector database alongside an…

16
r/LocalLLaMA community 11d ago

FP8/BF16 users seeing someone run IQ1_S

https://preview.redd.it/egchx5bap8th1.png?width=746&format=png&auto=webp&s=ab25375dcb6c204400ac8ba9b04497ded4c34e25 Why FP8/BF16 get horrified/disgusted even with FP4/INT4 or Any 4 bit quantisation method?   submitted by   /u/zyxciss [link]   [comments]

10
r/LocalLLaMA community 11d ago

Aleph-Alpha/Kolibri-1 · Hugging Face - 78B parameters. 3.46B active. Up to 1M tokens of context - Apache 2.0

From Aleph Alpha on 𝕏: https://x.com/Aleph__Alpha/status/2106306840657297814 Tech report: https://aleph-alpha.com/downloads/tech-report.pdf   submitted by   /u/Nunki08 [link]   [comments]

15
r/LocalLLaMA community 11d ago

Anyworld, a self-hosted multiplayer text RPG where a local LLM is the Dungeon Master

Hey everyone, I’ve been working on a game called Anyworld. It’s a browser-based multiplayer (single player also supported) text adventure inspired by the early days of AI Dungeon, especially its browser-based free version AI Dungeon 2. The setup is pretty straightforward: one…

28
r/LocalLLaMA community 11d ago

Fixed long-horizon task drift on local setups using a deterministic state plugin

Ran into an annoying issue with local models on long tasks. Once context window compaction hits after a few thousand tokens, the model loses sight of the original scope. Even with good system prompts, a few compaction cycles cause goal drift, hallucinated task completion, or…

10
r/LocalLLaMA community 11d ago

The ultimate guide to multi-harness RL

Hi folks, it's Lewis here from the post-training team at Hugging Face. We've been exploring how to train open models in different coding harnesses and wrote up a looong guide on how we solved this using open source libraries like TRL and the Harbor framework for RL environments.…

23
r/LocalLLaMA community 11d ago

Local Ai Pc 7663 Dual Epyc / Dual 9709

CPU Information Name AMD EPYC 7663 Topology 2 Processors, 112 Cores, 224 Threads Memory Information RDiMM 2933Mhz Size 1007.61 GB System Information Operating System Ubuntu 24.04.5 LTS Motherboard Giga Computing MZ72-HB2-00 GPUs 2x ASUS Turbo R9700 AI Pro 32 GB newest BIOS low…

11
r/LocalLLaMA community 11d ago

What are you expectations from Kimi K3.5?

Kimi K2 was already good but they took K2.5 a whole new level with so much of their continual learning phase, I believe it was on more 20-25T tokens iirc. Similarly K3 is just such an amazing model, I just love this model, wondering how amazing K3.5 will be!!   submitted by…

27
r/LocalLLaMA community 11d ago

gufo_windows pre-package for Strix Halo users

In the latest release of the completely unofficial gufo port to windows I've added a pre-packaged library that you can use to try out gufo for yourself. No need to build anything just grab the .zip and unpack it. I've also added start.cmd for easily starting the server, it will…

4
r/LocalLLaMA community 11d ago

I tried building a small RAG search node for Qwen3.8 27B using a fake AliExpress Mini PC... and Intel sent me back to 2018.

I love Qwen3.8 27B so much that I decided to show my gratitude to the Alibaba ecosystem by building a dedicated RAG/search node using a cheap Mini PC from AliExpress. Turns out, my ecosystem loyalty got rewarded with an absolute masterpiece of fraud: Promised: Intel N150 +…

15
r/LocalLLaMA community 11d ago

LiquidAI/LFM2.5-Encoder 250M/350M

LFM2.5-Encoder-350M is a multilingual bidirectional encoder built on the LFM2 architecture — a larger encoder for maximum downstream quality. It is a masked language model with full bidirectional attention, designed to be fine-tuned into task-specific models (classification,…

8
r/LocalLLaMA community 11d ago

Qwen3.8-Flash-Next 177B running at 11–15 tok/s on a single RTX 5070 12GB + 32GB RAM DDR4

Benchmarking an LLM here with a NVIDIA RTX 5070 12 GB VRAM here I had been working on a llama.cpp based expert streaming setup for Qwen3.8-Flash-Next 177B (UD-IQ3_XXS) on Windows. Benchmark is about 11.5 tok/s, up from roughly 7 tok/s on the inherited setup. In normal…

35
r/LocalLLaMA community 11d ago

qwen4exp : halve the indexer score memory by ServeurpersoCom · Pull Request #29825 · ggml-org/llama.cpp

Qwen Flash Next now uses less VRAM   submitted by   /u/jacek2023 [link]   [comments]

25
r/LocalLLaMA community 11d ago

gufo-Qwen3.6-35B-A3B-Q6dense - 3095tok/s prefill; 190 tok/s decode on Strix Halo

  submitted by   /u/nubela [link]   [comments]

6
r/LocalLLaMA community 11d ago

Has anyone noticed this trend toward writing/speaking style among newer models (both open and closed models). They are trending toward information density and expanded vocabulary. It's not quite 'caveman speak' but trending that way.

I've noticed a trend with most new models with regards to their writing style. They are creating a new style, and this seems common among them. It's very information-dense. Here is an example from GLM 5.3 Flash. I'm gonna be honest here and say that my prompt was kinda silly; my…

15
r/LocalLLaMA community 11d ago

mlsubgen — subtitles in 45 languages for your videos, entirely on your own machine

Full disclaimer: I've leaned heavily on Fable to develop this, but I've tested it thoroughly on my own library for a couple of months before putting it on GitHub. I live in Thailand, and it started as a way to get Thai subtitles for Shin-chan for Thai friends and for expat…

20
r/LocalLLaMA community 11d ago

Anyone sitting on a lot of slow system memory and a modest GPU.. try Strata + Qwen3.8 Next.

IQ3_XXS weights are just under 80GB and my slowww DDR4+7900XTX is stabilizing around 45-70/s (sometimes higher while coding depending on mtp). Looking online I'm seeing similar results for users with 12GB and 16GB cards, and significantly faster numbers for owners of DDR5. (In…

22
r/LocalLLaMA community 11d ago

The mines were hard on you.

Don't worry after some makeup and new fans you should feel better. And if you make it Mr.Hermes will treat you well.....   submitted by   /u/levoniust [link]   [comments]

4
r/LocalLLaMA community 11d ago

Could the EU Kids Act (or the like) be use potentially(Trojan horse) as excuse to go after the AI Open weights?

  submitted by   /u/UltraFOV [link]   [comments]

31
r/LocalLLaMA community 12d ago

I'm pretty close to the middle thanks to you all

It's been a blast and learning a ton. But seriously, you all have me down a rabbit hole that my wallet and hours of sleep need to be pulled out of.   submitted by   /u/pmarsh [link]   [comments]

33
r/LocalLLaMA community 12d ago

DataInventor: an open-source replication of Adoption Lab's Invent a Dataset

  submitted by   /u/asankhs [link]   [comments]

29
r/LocalLLaMA community 12d ago

Unitree just dropped UnifoLM-WLA-1.0 — a single 6B model that does 64 whole-body + tabletop tasks on a real humanoid

https://unigen-x.github.io/unifolm-wla.github.io/ Unitree Robotics released UnifoLM-WLA-1.0, their new general-purpose humanoid foundation model. Key points: • 6B parameters • Trained on ~2,500 hours of real robot data • One model handles 64 tasks (10 whole-body + 54 tabletop) •…

6
r/LocalLLaMA community 12d ago

So... Should we turn the page on this past week hype? or Do you have any success cases to inspire the rest?

  submitted by   /u/JLeonsarmiento [link]   [comments]

23
r/LocalLLaMA community 12d ago

I've ended up with an AI lab in a public community college. What should we actually be teaching?

Looking for some ideas from people who know a lot more about this than I do. We've got funding for a small AI lab in a public further education college in Ireland (roughly community college in the US). The hardware is reasonably decent. The goal is to give students useful skills…

4
r/LocalLLaMA community 12d ago

Peacebell - a from-scratch small language model

I've been using my free time during weeknights and weekends for the last 11 months working on and refining a small domain-specific language model. It specializes on information about World War II. The number one question I get asked about this is "Why did you pick World War II?"…

20
r/LocalLLaMA community 12d ago

I spent 3 weeks testing local Qwen3.8 on the new low-latency SGLang/vLLM recipes: DFlash2 2.8x. Builds: RadixArk + Inferact 27B NVFP4, 27B BF16, orcarouter 27B Uncensored, Flash-Next NVFP4

Hey guys, Last time I tested Qwen3.8-Flash-Next on its own. This time I put three Qwen3.8 checkpoints through the same 10 tests on the same RTX PRO 6000: RadixArk/Qwen3.8-27B-NVFP4 (dense) orcarouter/Qwen3.8-27B-Uncensored-NVFP4 (dense, uncensored fine-tune)…

20
r/LocalLLaMA community 12d ago

Buying RTX 5090 At Micro Center Reportedly Now Requires Paperwork, Including A No-Export Declaration

  submitted by   /u/Boomfrag [link]   [comments]

33
r/LocalLLaMA community 12d ago

microsoft/FrogNano-4B-2609 · Hugging Face

An agentic model from Microsoft for the GPU poor https://huggingface.co/bartowski/FrogNano-4B-2609-GGUF FrogNano is derived from Qwen/Qwen3.5-4B, a general-purpose post-trained model designed for language, reasoning, coding, agentic, and multimodal tasks. FrogNano inherits…

22
r/LocalLLaMA community 12d ago

Self-hosting AI does not save money, and I do it anyway

Hi folks, I'm a long-time lurker and big fan of this subreddit and a massive self-hosting fan (also outside of AI). I doubt many people will disagree with me here because I see the same arguments being made in many posts. However, I thought it might be interesting to share…

4
r/LocalLLaMA community 12d ago

CUDA: fuse shared experts into MMVQ by am17an · Pull Request #29184 · ggml-org/llama.cpp

MoE speedup, but only for some MoE architectures (like Qwen 35B A3B)   submitted by   /u/jacek2023 [link]   [comments]

22
r/LocalLLaMA community 12d ago

New Architecture from Percepta: Spotlight — separating intelligence from memory, allowing knowledge and skills to grow without changing the model's weights.

https://www.percepta.ai/blog/can-llms-grow-their-own-capabilities https://www.percepta.ai/blog/spotlight-memory "Our new architecture, Spotlight, replaces attention with a memory that escapes this trade-off: it is the first architecture to achieve infinitely growing memory…

23
r/LocalLLaMA community 12d ago

Heavily quantized Qwen3.8-Flash vs Q8 Qwen3.8-27B - thoughts?

I'm currently choosing between IQ3_XXS Qwen-3.8-Flash and Q8_0 27B. This month I don't have anything complex enough to justify either's potential so I've got a fairly bad read on how these two stack up in terms of intelligence. Have any of you compared the two enough to speak to…

33
r/LocalLLaMA community 12d ago

I made my iPhone a second GPU for my 24 GB MacBook: Qwen 3.8 27B prefills 29–44% faster & my holds part of the CTX window.

**DISCLAIMER** THE PREFILLING TPS SHOWN ON THE PHONE IS COMPUTED ONLY FOR THE LAYERS IT HOLDS. ALREADY FIXING IT TO SHOW END-TO-END PREFILL RATE. NUMBERS BELOW ARE ACCURATE FOR E2E PREFILL RATE. Every file or tool result my agent reads on a 24 GB M4 Pro MacBook is a wait, and…

33
r/LocalLLaMA community 12d ago

New 64GB DGX Spark. Significantly higher price for the original 128GB model $6,950 USD

  submitted by   /u/Norwood_Reaper_ [link]   [comments]

5
r/LocalLLaMA community 12d ago

Update #3: Post training yandex/AliceAI-80B-A3B [instruct!] from scratch

Last update for those who may be following: https://www.reddit.com/r/LocalLLaMA/comments/1wv5h8x/update_2_post_training_yandexaliceai80ba3b/ I screwed up guys 😂 Turns out my loss curve during my last run was legitimately unhealthy - as some of you, and myself, were concerned…

34
r/LocalLLaMA community 12d ago

RTX Spark laptops and mini desktops rumored to launch Oct 7th (24GB to 128GB variants possibly)

Basically a DGX Spark minus the ConnectX-7 ports. I’ve seen expected initial pricing from like $1800 to $2900. Not sure what configurations are actually at those price points. It’s all Internet hearsay until we actually see these things ship, but it’s nice to know that it’s…

30
r/LocalLLaMA community 12d ago

Qwen3.8-27B-Humanlike-Chat 2.0: texts like a human, now with tool calls and better instruction following

Last month I posted a Qwen3.8-27B LoRA that makes it talk like a person instead of an assistant. It got a lot more attention than I expected: 700+ upvotes, 248 comments and 44k downloads since. I read every comment. People really don't like assistant speak, so its tone of voice…

22
r/LocalLLaMA community 12d ago

Strata on a power limited 5090 and 96GB of DDR5-6400 is cranking out 150-200 tok/s decode and 5-6k prefill! Qwen3.8-Flash-Next at IQ3_S, CTX at 128k tokens (8-bit).

  submitted by   /u/z0_o6 [link]   [comments]

10
r/LocalLLaMA community 12d ago

AxiomicLabs' new benchmark, Tiny Theory of Mind, meant to gauge small models' theory of mind capabilities, has made it to the front page of Hugging Face's Datasets

  submitted by   /u/Megneous [link]   [comments]

17
r/LocalLLaMA community 12d ago

New in llama.cpp: Decision Models

  submitted by   /u/paf1138 [link]   [comments]

15
r/LocalLLaMA community 12d ago

Tuned/abliterated Qwen3.8-27b into a 24gb card 262k guff using the newest unreleased version of LexiPanel. It's fast with reliable draft acceptance. Made for 7900xtx but should work on whatever 24gb card with this setup and headless. Doesn't get dumber while coding like most of…

https://huggingface.co/Wa1k3r/Qwen3.8-27b-CODER-4q_xs-24GB-262k-Optimalcardfit Qwen3.8-27B CODER — IQ4_XS imatrix · 24 GB card fit · ~262k context · MTP draft Quantized, Abliterated, and fitted by LexiPanel. Its Fit planner chose the tensor mix to fill one 24 GB GPU at about…

30

I built a code knowledge graph tool that's actually MIT licensed (fully local, no cloud)

Sopro V2 Turbo 2610: cleaner cloned voices, same 120M model, same CPU speed

Two local Qwen ( 3.8 27b unsloth Q6 and Qwen flash next strata coder ) models vs Claude Opus 4.6 on the same 3 coding tasks. One of them tied it. Not here to start a fight, just sharing numbers

Come let your LLMs play World of Warcraft

POV 8GB VRAM + 16GB RAM folks checking locallama everyday:

Two ~300B MoE models, each on ONE 128 GB mini PC (AMD Strix Halo): GLM-5.3-Flash at ~580 tok/s prefill, MiMo-V2.6-Flash up to 44 tok/s decode. EXL3 weights + open ROCm engine

Yes bots we get it, Strata is good now please stop

Anyone using a local AI meeting notes setup instead of Fathom?

Qwen Flash next on 64GB RAM unified iGPU anyone ? (non-mac)

Replacing vector databases with SQLite and SIMD hypervectors in under 1.2GB VRAM (Hillock)

FP8/BF16 users seeing someone run IQ1_S

Aleph-Alpha/Kolibri-1 · Hugging Face - 78B parameters. 3.46B active. Up to 1M tokens of context - Apache 2.0

Anyworld, a self-hosted multiplayer text RPG where a local LLM is the Dungeon Master

Fixed long-horizon task drift on local setups using a deterministic state plugin

The ultimate guide to multi-harness RL

Local Ai Pc 7663 Dual Epyc / Dual 9709

What are you expectations from Kimi K3.5?

gufo_windows pre-package for Strix Halo users

I tried building a small RAG search node for Qwen3.8 27B using a fake AliExpress Mini PC... and Intel sent me back to 2018.

LiquidAI/LFM2.5-Encoder 250M/350M

Qwen3.8-Flash-Next 177B running at 11–15 tok/s on a single RTX 5070 12GB + 32GB RAM DDR4

qwen4exp : halve the indexer score memory by ServeurpersoCom · Pull Request #29825 · ggml-org/llama.cpp

gufo-Qwen3.6-35B-A3B-Q6dense - 3095tok/s prefill; 190 tok/s decode on Strix Halo

Has anyone noticed this trend toward writing/speaking style among newer models (both open and closed models). They are trending toward information density and expanded vocabulary. It's not quite 'caveman speak' but trending that way.

mlsubgen — subtitles in 45 languages for your videos, entirely on your own machine

Anyone sitting on a lot of slow system memory and a modest GPU.. try Strata + Qwen3.8 Next.

The mines were hard on you.

Could the EU Kids Act (or the like) be use potentially(Trojan horse) as excuse to go after the AI Open weights?

I'm pretty close to the middle thanks to you all

DataInventor: an open-source replication of Adoption Lab's Invent a Dataset

Unitree just dropped UnifoLM-WLA-1.0 — a single 6B model that does 64 whole-body + tabletop tasks on a real humanoid

So... Should we turn the page on this past week hype? or Do you have any success cases to inspire the rest?

I've ended up with an AI lab in a public community college. What should we actually be teaching?

Peacebell - a from-scratch small language model

I spent 3 weeks testing local Qwen3.8 on the new low-latency SGLang/vLLM recipes: DFlash2 2.8x. Builds: RadixArk + Inferact 27B NVFP4, 27B BF16, orcarouter 27B Uncensored, Flash-Next NVFP4

Buying RTX 5090 At Micro Center Reportedly Now Requires Paperwork, Including A No-Export Declaration

microsoft/FrogNano-4B-2609 · Hugging Face

Self-hosting AI does not save money, and I do it anyway

CUDA: fuse shared experts into MMVQ by am17an · Pull Request #29184 · ggml-org/llama.cpp

New Architecture from Percepta: Spotlight — separating intelligence from memory, allowing knowledge and skills to grow without changing the model's weights.

Heavily quantized Qwen3.8-Flash vs Q8 Qwen3.8-27B - thoughts?

I made my iPhone a second GPU for my 24 GB MacBook: Qwen 3.8 27B prefills 29–44% faster & my holds part of the CTX window.

New 64GB DGX Spark. Significantly higher price for the original 128GB model $6,950 USD

Update #3: Post training yandex/AliceAI-80B-A3B [instruct!] from scratch

RTX Spark laptops and mini desktops rumored to launch Oct 7th (24GB to 128GB variants possibly)

Qwen3.8-27B-Humanlike-Chat 2.0: texts like a human, now with tool calls and better instruction following

Strata on a power limited 5090 and 96GB of DDR5-6400 is cranking out 150-200 tok/s decode and 5-6k prefill! Qwen3.8-Flash-Next at IQ3_S, CTX at 128k tokens (8-bit).

AxiomicLabs' new benchmark, Tiny Theory of Mind, meant to gauge small models' theory of mind capabilities, has made it to the front page of Hugging Face's Datasets

New in llama.cpp: Decision Models

Tuned/abliterated Qwen3.8-27b into a 24gb card 262k guff using the newest unreleased version of LexiPanel. It's fast with reliable draft acceptance. Made for 7900xtx but should work on whatever 24gb card with this setup and headless. Doesn't get dumber while coding like most of…