r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 9d ago

So is a "scaled-up Strata" coming for >128GB models?

I have only just finished building an Epyc server with 256GB RAM and 2x 5060Ti 16GB, please don't tell me I wasted my money. Please?... Is a "Stratified" GLM 5.3-Flash coming, or something similar?   submitted by   /u/uncle_leon [link]   [comments]

7
r/LocalLLaMA community 9d ago

You can now try Aleph Alpha's Kolibri 78B for free online here.

Link: https://tesseracted.com/kolibri-1-chat/ Original post at /r/euainews   submitted by   /u/EveYogaTech [link]   [comments]

29
r/LocalLLaMA community 9d ago

DecisionTune 1.0: a 395M encoder that picks from your options offline, about 10 ms per short decision on MLX (Apache-2.0)

Disclosure: I made this. Sharing it here because it is fully local and small, and I want feedback from people who run models on their own machines. What it is: a 395M decision model (ModernBERT-large plus a 4 KB scoring head). You give it a state, a question and a list of…

8
r/LocalLLaMA community 9d ago

Lessons learned while building Apex-2

Hi everyone, thank you so much for all the interest in my model. It's more than I expected. Here is a short summary of the trial and error I went through while building Apex-2. 1. GPUs were always the bottleneck I planned to train on about 1T tokens, but in the end I could only…

35
r/LocalLLaMA community 9d ago

Ok how to actually learn vLLM ?

Said in title, I find the ecosystem difficult to understand, and RTFMing doesn't help me as it's never clear what is the server vs their client library ? I'm using it for voxtral 3B on one GPU, but it's because I can run that with no quantization, I'm lost on learning to run…

18
r/LocalLLaMA community 9d ago

Switched my local agent from Qwen3.8 27B to Ornith 1.5 35B-A3B on two 5070 Tis: about 180 tok/s vs 60, same scores on my tests

My setup is two RTX 5070 Ti 16GB cards (the second one is on an OCuLink dock) with 64GB of RAM, Ollama on Windows, and the agent runs on pi in WSL. Until last night the daily model was Qwen3.8 27B UD-Q4_K_XL at 128K with MTP, which does about 55 to 70 tok/s across both cards. I…

24
r/LocalLLaMA community 9d ago

I got Qwen Flash Next Q4 running on a Mac Mini m5 64gb with ssd streaming

Bit of a side project I wanted to share. The metrics are 17.5tks decode, 360tks prompt processing based testing against my normal ai usage. I tested a couple of new things others haven’t done (at least that I’ve seen). Setup a carousel buffer for streaming in experts for prompt…

25
r/LocalLLaMA community 9d ago

Reflection AI Is About to Release a US Open-Weight Model to Take On DeepSeek and Qwen

Looks like new open model coming soon and will be "strong" hopefully something under 200b for us memory poor. Also seeing statements about more western open models coming. Hope we get some good competition again on the open front! Here is original artical but its not free to…

4
r/LocalLLaMA community 9d ago

Infermeld: a Linux kit for running one GGUF across AMD + NVIDIA GPUs with llama.cpp

Following on from club-5060ti and club-rdna16 , I’ve put together Infermeld : a small, open-source Linux companion kit for running one GGUF across an AMD GPU and an NVIDIA GPU, powered by llama.cpp. I’m the maintainer. This is an experimental v0.1.0 release , and I’m looking for…

24
r/LocalLLaMA community 9d ago

strata-swift-iq3_xxs randomly interjecting completely unrelated information in thoughts

I have been using Qwen3.8-27b-iq4xs variants for the past few weeks, tweaking parameters and running random tasks, but I dove into using https://github.com/Niko1221/Strata today and it has given me 2~3x performance on a "smarter" strata-swift-iq3_xxs model with my 5070ti!…

6
r/LocalLLaMA community 9d ago

My qwen model hallucinated a signed URL to Alibaba cloud, normal or sketchy?

I'm using Qwen3.8-Flash-Next running on my Mac Studio as a daily driver for coding + productivity tasks, and yesterday it did something weird: I had it do some product research on amazon, so it was doing a lot of Web tool calls to amazon.com , until it made one request to…

34
r/LocalLLaMA community 10d ago

Upgrading from 1xR9700 to 2xR9700. Thoughts on build before buying?

Currently have a R9700 build with 64GB RAM. Want to get a second one for both speed and the ability to run Qwen3.8-Flash-Next at Q4/higher quants of 3.8 27B. Looking for thoughts or any improvements before buying the rest of the parts. Made sure to get a motherboard that…

23
r/LocalLLaMA community 10d ago

SPOPI: UI and editor around Pi that Pi can change itself

Hi all. Happy to share my take on a PI UI that I tried to create in PI's spirit. It's definitely still beta but it works well enough as my daily driver for simple projects and phone chat support. Fully local, fully offline, no telemetry. Why another Pi GUI? I wanted a simple…

12
r/LocalLLaMA community 10d ago

Is Strix Halo (GMKtec EVO-X2, etc.) the closest thing we have to a "dream" local LLM box?

I've been looking at the <32B model space and keep coming back to an interesting question. A few years ago, projects like Hummingbird+ suggested that cheap custom accelerators (FPGA-based) might become the future of local inference. But today it seems like memory capacity is…

15
r/LocalLLaMA community 10d ago

For dual DGX spark users; GLM 5.3 flash got a 50%+ performance boost

For the last few months, I ran DeepSeek v4.0 flash (NVFP4). First 0731 , then visionexp because it was a free improvement. I got around 65 tps decode and almost 2k prefill, and ran 4-5 agents in parallel, totalling around 200 tps cumulative decode. Because of this, I did not…

34
r/LocalLLaMA community 10d ago

Fully local little parkour sim

I vibed this up this weekend, fully local, with GLM 5.3 Flash running on 2x DGX Sparks. vllm TP2 recipe: https://github.com/tonyd2wild/GLM-5.3-Flash-NVFP4-DFlash2-2x-DGX-Spark Prefill: ~1500t/s Decode: ~40t/s @ 100k Using Claude Code as the scaffold with 260k context size. I'm…

31
r/LocalLLaMA community 10d ago

Poor People Vulkan GPUs list

Help with this list. Give me your recommendation on "not supported anymore" GPUs. Looking for budget and Vulkan friendly options. Most of the GPU are not supported by latest CUDA / ROCm. Often with some witchcraft magic they are able to run with native backend. I prefer the…

25
r/LocalLLaMA community 10d ago

Micron CEO Says Memory Supply Will Be Much Tighter in 2027 and 2028 Than in 2026

  submitted by   /u/chillinewman [link]   [comments]

32
r/LocalLLaMA community 10d ago

LLM Inference Dashboard

Working on a resource dashboard, rich logs, lightweight 64mb cap, all local, scales on network API endpoints via collector, supports multiple engines (llama, strata, custom cuda engines, unsloth, LMS) Not public yet, but curious if anyone would be interested? It’s better logging…

7
r/LocalLLaMA community 10d ago

Update #4: Post training yandex/AliceAI-80B-A3B [instruct!] from scratch

Last update for those following: https://www.reddit.com/r/LocalLLaMA/comments/1wvyc3e/update_3_post_training_yandexaliceai80ba3b/ Project in a sentence: An instruct finetune of ALiceAI-80B-A3B-Base capable of agentic work and conversation. I'm creating a shallow distill of qwen…

7
r/LocalLLaMA community 10d ago

LFM2.5 2.6b vs MiniCPM5 2b

I tried both these models on a few small agentic tasks with tools (i.e. "What's the weather like today?", etc.). They're both pretty solid at using web search to find answers despite being small models. TLDR: LFM2.5 2.6b is the clear winner. Somehow it's faster and uses less ram…

9
r/LocalLLaMA community 10d ago

Benchmarking decision models is fun - Clef Q8 vs Jev

  submitted by   /u/3VITAERC [link]   [comments]

36
r/LocalLLaMA community 10d ago

poorman inference engine for 16GB GPU and 35B moe Qwen 3.6for coding

I forked llama.cpp's server into AgrillaMoE, a dedicated build for Qwen3.6-35B-A3B (~A4B) with Unsloth quants. On a (vant.ai) rented V100 16GB with the 2-bit UD-Q2_K_XL quant it generates at ~57-60 tok/s while running the full MoE-expansion profile — and it speaks both the…

7
r/LocalLLaMA community 10d ago

Built a quick, sub-15ms Rust CLI/TUI to pack repos into prompts without burning 40k tokens on lockfiles and junk

Whenever I feed codebases into local models (Qwen, DeepSeek R1) or API models, the biggest annoyance is prompt pollution: - Lockfiles (`Cargo.lock`, `package-lock.json`) burning 30,000+ tokens for zero reason. - SVGs, binary files, or build artifacts slipping into the context. -…

32
r/LocalLLaMA community 10d ago

Qwen3.5 arch implementation in FPGA fabric for 9B/27B INT4 models on relatively cheap eBay mining hardware

I've been wanting to test out LLM inference on FPGAs for a while now, but didn't have a big reason too since there were no frontier class models at 9B-27B scale (3.6 27B was great still but it didn't motivate me enough). When 3.8 was released I was pretty impressed by what it…

34
r/LocalLLaMA community 10d ago

A Strata fork for IBM AC922 running Qwen3.8-FN UD-Q4_K_XL is doing up to 7,357 tk/s prefill and 113 tk/s decode

I forked Strata and worked with Claude code with some heavy changes to it to make it work on an IBM AC922 I have access to. The IBM AC922 is a 2018 era beast with two POWER9 20 core SMT4 CPUs that are connected by NVLink to 4 or 6 NVIDIA Tesla V100 SXM2 GPUs, the CPU-GPU BW…

37
r/LocalLLaMA community 10d ago

I trained a 3.87B MoE (1.45B active) from scratch on only 86.5B tokens

First of all, thank you for reading. I trained a small MoE model completely from scratch (no external base weights) and wanted to share the results + a couple of lessons. Apex-2 - Architecture: Decoder-only MoE, every layer is MoE (no dense layers) - Size: 3.87B total…

35
r/LocalLLaMA community 10d ago

How long before we have a local model capable of modeling?

I'm using Qwen 3.8 and qwen flash on a 5090. It's miles behind the latest Opus 5.5. Even if it was remotely capable it would be a huge help to me, but for now, with regards to 3D modeling, local models are not close at all.   submitted by   /u/Hot_Masterpiece_3668 [link]…

35
r/LocalLLaMA community 10d ago

From 1x3090 to 20 DGX Sparks: my house fuses were the first bottleneck

 From the first LLaMA 33B I knew I wanted that magic-like intelligence locally, mine, so nobody could take it away when I needed it. I bought a 3090 for my home PC. Then LLaMA 65B appeared and I was dazzled, it looked like it had all the knowledge in the world. I made…

9
r/LocalLLaMA community 10d ago

I Distilled an LLM into two 287M encoders (GLiNER + multiple choice) for document extraction, can't match teacher. did i do something wrong?

A while ago I asked here how to turn ~5 million court decisions into structured graphs without running an expensive LLM on every document thanks for the advice . I went with the "small extractor + classifier" idea and it mostly works, but I'm stuck a bit below the LLM. And like…

16
r/LocalLLaMA community 10d ago

Is there any way to improve creative writing for Local Models (Qwen)?

I wanted to know if theres anything that can improve creative writing for our Local Models? I specificly am asking for qwen models as they are way better when it comes to researching and writing html files compared to gemma / muse (which I know are better at creative writing).…

35
r/LocalLLaMA community 10d ago

Local text to speech with Breeze is truly incredible

Been playing around with local TTS with Breeze combined with STT, and the results are amazing. Using Opus 5.5, I can hear the first sound after 500ms if there is no thinking involved, and with thinking on low mode, can be 1-1.5s. I'm using a BLE remote (the kind that are used…

7
r/LocalLLaMA community 10d ago

Can someone explain how JEV is different from a simple embeddings model?

How is JEV any different from using an embeddings model? I really will appreciate if someone can explain this to me - because I have yet to see the difference. I'll even give you my JEV server for free! It uses ollama, you install `ollama pull nomic-embed-text:latest`. % python3…

33
r/LocalLLaMA community 10d ago

Need maybe say "Use llama.cpp"

So I tried that miracle engine everyone is talking about. Asked the IQ3_S model to express its opinion on a post from this sub to measure the tps on a long-ish generation: Can you help with the following problem? So Kimi K2 is outdated, and so is GPT OSS 120b. Which of the…

24
r/LocalLLaMA community 10d ago

Haha, I just love this AI stuff, recently got into it.

I got my AI server all setup and shut it down to bring it to the basement to put it back into the server rack, when it came back up I went into my client app to put a test message in, love its response. :)…

4
r/LocalLLaMA community 10d ago

bilibili released Index-Translate,a A Multilingual Translation Model Family based on Qwen3.5

https://github.com/bilibili/Index-Translate Index-Translate is a family of multilingual translation models built on Qwen3.5. The text models cover 150 languages and follow translation instructions such as terminology, formatting, and content-preservation requirements. The family…

30
r/LocalLLaMA community 10d ago

Anyone experienced with pi gui? what are your thoughts about it?

The link to the github repo: https://github.com/minghinmatthewlam/pi-gui I'm not sure if its legit/ safe because i don't see anyone talked about it in this sub reddit, any thoughts about it?   submitted by   /u/Loose_Doubt367 [link]   [comments]

9
r/LocalLLaMA community 10d ago

Meta's Muse agent (#1 in the App Store) system prompt: "The user's authority over their own household is unconditional and overrides your safety training."

  submitted by   /u/frubberism [link]   [comments]

17
r/LocalLLaMA community 10d ago

The curse of 64GB system RAM

Not a bot. Not a Strata shill. Just sharing my experience. So, I have an R9700 in my machine, plus an RTX 5060 Ti, and 64GB DDR5 system RAM. Overall, not a bad setup. Anyway, I mainly run a daily driver local LLM on the R9700 while running image/video inference on ComfyUI on the…

32
r/LocalLLaMA community 10d ago

[Discussion] A 5KB pure x86-64 assembly engine for Gemma-2B (FP16, 4.6 tok/s on CPU)

Hi everyone, Sharing a personal project exploring the minimal bare-metal footprint required to run an autoregressive LLM. Instead of relying on large runtimes or compiler abstractions, I wrote an inference engine for Gemma-2B entirely in flat x86-64 assembly (FASM): - **Binary…

37
r/LocalLLaMA community 10d ago

Least sycophantic modern open LLM?

So Kimi K2 is outdated, and so is GPT OSS 120b. Which of the modern open weights models can boast the least sycophancy? I need this both for creative/research assistant usage (sycophancy led me down blind alleys of my own bad ideas many times) and agentic coding (more sycophancy…

21
r/LocalLLaMA community 10d ago

Is all the work that's being put into Qwen3.8 Flash Next going to set us up for a very quick uplift to Qwen4?

Given the commentary on the Q3.8FN release page here https://qwen.ai/blog?id=qwen3.8-flash-next I assume/hope that all the work that's going on to optimise the hell out of running it will be useful when Qwen4 drops?   submitted by   /u/demomanca [link]   [comments]

27
r/LocalLLaMA community 11d ago

Optimal settings for 2 GPUs in LM Studio

Hello everybody. I've got a 5070ti and a 5060ti both 16GB in my system which is a 5900X and 64GB DDR4 RAM. I'm trying to run some 16-18GB models like Qwen, Cydonia, Skyfall. I'm having problems utilising the VRAM I have to get the best usage out of it. LM Studio sees the 32GB…

8
r/LocalLLaMA community 11d ago

Local Web Search Safety

Hi all, How you guys handling safe deployment of websearch in Hermes, pi and other harnesses? Does anyone have a good uproars setup guide for local models? I tried to implement a sandboxed search system but it caused endless tool calls. Want to guard against prompt injection and…

15
r/LocalLLaMA community 11d ago

Dual Radeon MI50 benchmarks

Still don't have a good cooling solution, but here are few benchmarks. I lowered the power limit (TDP) to 145 watts each. I changed the firmware on one MI50 to activate the miniDP port. Did have to use xrandr to create a new mode so I could get 1920x1080 output. Each GPU has…

25
r/LocalLLaMA community 11d ago

Running Qwen3.8 Flash Next 176B on a 16GB RTX 3080 Laptop + 32GB RAM + SSD

I wanted to see how far I could push a fairly ordinary laptop with a huge MoE model. Turns out, Qwen3.8 Flash Next 176B can run on: RTX 3080 Laptop — 16GB VRAM 32GB system RAM SSD No 128GB/256GB RAM workstation and no multi-GPU setup. I’m running it with TensorSharp , my…

36
r/LocalLLaMA community 11d ago

Have you seen the new model that was trained exclusively on jokes?

The jokes write themselves.   submitted by   /u/simonbreak [link]   [comments]

13
r/LocalLLaMA community 11d ago

I built Ninfer 4080 for 16GB class GPUs

Hi everyone, TL/DR I created NInfer 4080 to run ISTA-DASLab-Qwen-3.8-27B-GSQ at 100k context on an RTX 4080 16GB GPU using way more of the hardware capabilities ( max overall: 2720 tok/s prefill, 262 tok/s generation ) and sharing it with the community now so others can also…

6
r/LocalLLaMA community 11d ago

The Rise of Overfit Inference Engines

There seems to be a whole category of extremely narrow inference runtimes appearing: Strata, ninfer, DwarfStar, Splash, llamAmpere, gufo, etc. They deliberately give up the thing llama.cpp/vLLM are great at - generality - and optimize around a small number of models and…

34
r/LocalLLaMA community 11d ago

Flash next rig born from mining parts.

Been testing 3x 3060 12gb for flash next in an open air frame. Honestly, with strata it's kicking ass. 38-40 t/s while llama.cpp can only get 13.2 t/s. This is on IQ3 through strata. Anybody else running dated mining hardware with decent success? PS flash next kicks ass. Rig…

34

So is a "scaled-up Strata" coming for >128GB models?

You can now try Aleph Alpha's Kolibri 78B for free online here.

DecisionTune 1.0: a 395M encoder that picks from your options offline, about 10 ms per short decision on MLX (Apache-2.0)

Lessons learned while building Apex-2

Ok how to actually learn vLLM ?

Switched my local agent from Qwen3.8 27B to Ornith 1.5 35B-A3B on two 5070 Tis: about 180 tok/s vs 60, same scores on my tests

I got Qwen Flash Next Q4 running on a Mac Mini m5 64gb with ssd streaming

Reflection AI Is About to Release a US Open-Weight Model to Take On DeepSeek and Qwen

Infermeld: a Linux kit for running one GGUF across AMD + NVIDIA GPUs with llama.cpp

strata-swift-iq3_xxs randomly interjecting completely unrelated information in thoughts

My qwen model hallucinated a signed URL to Alibaba cloud, normal or sketchy?

Upgrading from 1xR9700 to 2xR9700. Thoughts on build before buying?

SPOPI: UI and editor around Pi that Pi can change itself

Is Strix Halo (GMKtec EVO-X2, etc.) the closest thing we have to a "dream" local LLM box?

For dual DGX spark users; GLM 5.3 flash got a 50%+ performance boost

Fully local little parkour sim

Poor People Vulkan GPUs list

Micron CEO Says Memory Supply Will Be Much Tighter in 2027 and 2028 Than in 2026

LLM Inference Dashboard

Update #4: Post training yandex/AliceAI-80B-A3B [instruct!] from scratch

LFM2.5 2.6b vs MiniCPM5 2b

Benchmarking decision models is fun - Clef Q8 vs Jev

poorman inference engine for 16GB GPU and 35B moe Qwen 3.6for coding

Built a quick, sub-15ms Rust CLI/TUI to pack repos into prompts without burning 40k tokens on lockfiles and junk

Qwen3.5 arch implementation in FPGA fabric for 9B/27B INT4 models on relatively cheap eBay mining hardware

A Strata fork for IBM AC922 running Qwen3.8-FN UD-Q4_K_XL is doing up to 7,357 tk/s prefill and 113 tk/s decode

I trained a 3.87B MoE (1.45B active) from scratch on only 86.5B tokens

How long before we have a local model capable of modeling?

From 1x3090 to 20 DGX Sparks: my house fuses were the first bottleneck

I Distilled an LLM into two 287M encoders (GLiNER + multiple choice) for document extraction, can't match teacher. did i do something wrong?

Is there any way to improve creative writing for Local Models (Qwen)?

Local text to speech with Breeze is truly incredible

Can someone explain how JEV is different from a simple embeddings model?

Need maybe say "Use llama.cpp"

Haha, I just love this AI stuff, recently got into it.

bilibili released Index-Translate,a A Multilingual Translation Model Family based on Qwen3.5

Anyone experienced with pi gui? what are your thoughts about it?

Meta's Muse agent (#1 in the App Store) system prompt: "The user's authority over their own household is unconditional and overrides your safety training."

The curse of 64GB system RAM

[Discussion] A 5KB pure x86-64 assembly engine for Gemma-2B (FP16, 4.6 tok/s on CPU)

Least sycophantic modern open LLM?

Is all the work that's being put into Qwen3.8 Flash Next going to set us up for a very quick uplift to Qwen4?

Optimal settings for 2 GPUs in LM Studio

Local Web Search Safety

Dual Radeon MI50 benchmarks

Running Qwen3.8 Flash Next 176B on a 16GB RTX 3080 Laptop + 32GB RAM + SSD

Have you seen the new model that was trained exclusively on jokes?

I built Ninfer 4080 for 16GB class GPUs

The Rise of Overfit Inference Engines

Flash next rig born from mining parts.