r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 12d ago

Two 96 GB Ascend cards crun Qwen3.8-flash-next hardware notes, vLLM work, benchmarks, and what is next

I have been building a somewhat unusual local inference machine around two Huawei Atlas 300I Duo cards. They are relatively inexpensive, passive, dual-accelerator PCIe cards with 96 GB of device memory apiece. They are also absolutely not drop-in CUDA replacements. When I first…

36
r/LocalLLaMA community 12d ago

Inference Engineering for Dummies

Hi all! I am a former SWE who has recently transitioned into inference engineering. I launched a side hustle a few months back and I've just taken it full time due to excessive demand. Its been such an opportunity because the people optimizing runtime are far fewer than the…

20
r/LocalLLaMA community 12d ago

I made my own cybersecurity benchmark and ran Qwen3.8 27B, here's how a local model actually does at hacking

Hey local AI community, I've been working on this for a while and finally feel ok sharing it. It's a cyber benchmark where the model gets a shell in an isolated docker box and has to find the exact flag. Pwn, web, crypto, rev, forensics, a few real CVEs and some multi-stage…

4
r/LocalLLaMA community 12d ago

Apex Compute developing open-source Mesa Vulkan driver for their hardware

  submitted by   /u/Fcking_Chuck [link]   [comments]

11
r/LocalLLaMA community 12d ago

Qwen3.8-Flash-Next (5.05bpw + ngram at bf16) exl3 on one r9700: 863 t/s prefill and 35 t/s decode at 230k context (256k max), is that ok or am i missing something?

Hi, i got QFN running on my single r9700 but im not sure if i did everything right to get the best quality and speed out of this setup. Dont want to annoy anybody, maybe someone with the same card can tell me if this looks normal. What i run: - model: Qwen3.8-Flash-Next from…

31
r/LocalLLaMA community 12d ago

llama, server: add /v1/systemone API (models: laya, julia-1, lev, openjev, kev) by ngxson · Pull Request #29818 · ggml-org/llama.cpp

Now you can jev without jev https://huggingface.co/ggml-org/OpenJev-GGUF https://huggingface.co/ggml-org/lev-GGUF https://huggingface.co/ggml-org/Julia-1-GGUF https://huggingface.co/ggml-org/Laya-GGUF https://huggingface.co/ggml-org/Kev-4B-GGUF   submitted by  …

25
r/LocalLLaMA community 12d ago

Does anyone know if any new releases from Mistral are planned?

It’s been a long time since the last models came out. I notice they are selling GLM on the site, and I wonder if they are developing something, given the long silence.   submitted by   /u/LegacyRemaster [link]   [comments]

4
r/LocalLLaMA community 12d ago

best <40B alternatives to Qwen/Deepseek for (1) Coding (2) Long document QA test

Due to some reasons, Qwen/Deepseek Chinese models are NOT allowed in the my workplace. So, what local models, do you think, is the best alternatives to Qwen/Deepseek for (1) Coding (2) Long document QA test (like giving a long medical history of 120k tokens and ask a question…

23
r/LocalLLaMA community 12d ago

Mirostat?

Reading another post made me think: Is anybody still using Mirostat? It was all the rage and people said it avoided the “boredom trap” for long texts. Do newer generation models not need that anymore, or are other samplers superior?   submitted by   /u/ParvusNumero…

19
r/LocalLLaMA community 12d ago

Detecting hallucinations in local models without eating VRAM: What we learned testing 1.5B to 120B models

Hey everyone, If you run local models via Ollama in production or personal projects, you've probably run into the hallucination problem: how do you know when a model is hallucinating without burning extra VRAM or waiting 5 seconds for a heavy judge model? The standard academic…

8
r/LocalLLaMA community 13d ago

Update on the “Monstrosity”. 6 BC-250 board cluster

This is 6 bc-250 ex mining boards with 5 in the asrock 4u12g case they came in. After a lot of testing my current preferred setup is 4 boards running Qwen Next Flash IQ2_XS at 100k context with around 28 tok/s for short generation and 24 tok/s at 50k with around 115 ppt. The…

15
r/LocalLLaMA community 13d ago

Introducing ModelHarbor, a self-hosted model library

  submitted by   /u/toks-love [link]   [comments]

5
r/LocalLLaMA community 13d ago

I "built" a local LLM radio like the Pharaohs built the pyramids

My local stack is a couple DGX Sparks, a 5090, and a 4070 TI. This radio site took three local LLMs, and we generate music, voices, and wacky sound effects. I give you https://pilgrim.farm It's a lot more farm than pilgrim.   submitted by   /u/jwhh91 [link]  …

25
r/LocalLLaMA community 13d ago

Perplexity Decider 27B: Open weights decision model fine tune of Qwen3.8 27B

  submitted by   /u/rm-rf-rm [link]   [comments]

37
r/LocalLLaMA community 13d ago

Pi 1.0 released - MCP support now included by default

  submitted by   /u/psychohistorian8 [link]   [comments]

14
r/LocalLLaMA community 13d ago

Gufo performance .... 70tps Qwen 3.8 27b but you need to read the fine print.

So everyone has been yelling about how I should be using Gufo instead of halogen because it's open source and it's "just as good or better". Checking in on their GitHub (GitHub.com/gufo-org/gufo) got me immediately .. "Qwen 27B Q4: 70.56 tok/s single user, 123 tok/s with 8…

19
r/LocalLLaMA community 13d ago

What local AI model is good for game decomps/recomps?

Hello guys. Recently, there has been a boom in game decomps and recomps thanks to AI. If you look at the r/decomps and r/recomps subreddits, you can see it. They mostly seem to be using Claude or Codex.I wonder if it would be possible to do something similar with a local AI…

28
r/LocalLLaMA community 13d ago

DDR4/PCIe4 vs DDR5/PCIe5 for LLMs- I benchmarked them for pre-training. What are your thoughts?

Hello everyone. I've recently ran some experiments comparing DDR4/PCIe4 and DDR5/PCIe5 for AI workstations on a pre-training run, and would like to hear yours thoughts. First of all, and as a summary of my results ( code here:…

8
r/LocalLLaMA community 13d ago

Running 95.5 GiB Qwen3.8-Flash-Next at 41–52 tok/s on a 64GB Mac (1.76x faster than llama.cpp): Slipstream release, 130k context scaling, + Swift variant

I've been working on getting the 95.5 GiB Qwen3.8-Flash-Next model to run fast on a single 64GB Mac. In my earlier posts, I shared a custom expert-streaming fork of llama.cpp . It worked, but decode capped out around ~23–27 tok/s and slowed down as context grew. Today I'm…

5
r/LocalLLaMA community 13d ago

AMA about K2 Horizon, Meet our team from IFM

Hi r/LocalLLaMA! We’re researchers at the Institute of Foundation Models (IFM), an AI research lab dedicated to open and independent development of frontier-class foundation models. We recently released K2 Horizon a connected fleet of six fully open models with size ranging from…

6
r/LocalLLaMA community 13d ago

Now which lab is behind fledge alpha?

New free/trial model dropped on OC. Details are yet to be provided. Could this be deepseek v4.1 pro?   submitted by   /u/Gohab2001 [link]   [comments]

16
r/LocalLLaMA community 13d ago

What's the best setup for Qwen3.8 27b for a 16 gig VRAM?

Hello guys! I have been experimenting with qwen 3.8 for a long time and I hadn't been able to get reasonable speed. I am on a 5060Ti 16 GB, and although this gpu can game, I am aware that AI demands more than 16 GB. I am on a Fedora 44, AMD Ryzen 9600x and 16 GB system ram (16…

36
r/LocalLLaMA community 13d ago

Update #2: Post training yandex/AliceAI-80B-A3B [instruct!] from scratch

Last update: https://www.reddit.com/r/LocalLLaMA/comments/1wu9ksu/update_yandexaliceai_80ba3b_fine_tune_progress/ - basically, an instruct fine tune on the base model using a synthetic distilled data set. I've been posting regular updates so I imagine at least a few people have…

11
r/LocalLLaMA community 13d ago

Clef: Open Weights decision model by Cloudflare

  submitted by   /u/paf1138 [link]   [comments]

25
r/LocalLLaMA community 13d ago

Heretic is on PewDiePie!

So I haven’t played a computer game in 20 years, and I know nothing about Minecraft, and I definitely prefer classical literature over YouTube culture, but even I have heard about the individual called PewDiePie, for two reasons: His monicker starts with the initials of my own…

21
r/LocalLLaMA community 13d ago

Unsloth, Swift1.5, Peculiar-Ragdoll, ThinkingCap - Qwen3.8-27B

In a previous post I shared comparison between Swift1.5 and peculiar-ragdoll's checkpoints. Added the original unsloth Q4_K_XL and ThinkingCap Q4_K_M (they don't offer L or XL) to the comparison. Here are the results over a 69 set of eval questions. All tests are now run at same…

13
r/LocalLLaMA community 13d ago

Pi extension: Skip reasoning with local Qwen 27B and proceed to answer right now

pi-llama-skip-reasoning is an extension for the Pi.dev harness that forces a local llama.cpp model to stop reasoning and answer / act immediately. When you are deep into the ctx session and ask 27B a simple question about a fact or need a direct action, the model may still feel…

32
r/LocalLLaMA community 13d ago

We benchmarked 18 RAG pipelines against an agent loop on Google's FRAMES. The best pipeline hit 78.9%. The agent loop hit 92.7%.

Hybrid search, reranking, query decomposition, and query expansion are often treated as must-haves for good RAG. We wanted to see how much each actually helped, so we tested them. Same model, same embeddings, same documents, across all 824 multi-hop questions in FRAMES. We built…

32
r/LocalLLaMA community 13d ago

China and the memory market

Once china sets its goals for dominating a market it wins. Usually takes many years, but it happens. Can't compare the political will of a country versus profit and loss thinking of a corporation. China will eventually win in the memory market and current memory makers are at an…

35
r/LocalLLaMA community 13d ago

Jeff-Qwen3.5-0.8B v1.2 + 9 LoRA adapters: put it in front of Qwen3.8-27B for 38× faster decisions and +8.7 points accuracy, for under 2 GB extra memory

A few days ago I released Jeff-Qwen3.5-0.8B, a small "System 1" model that picks between options you define and returns a calibrated probability for each, in one forward pass. Speed was great on my M4 Max and RTX PRO 6000, but as a general zero-shot classifier it trailed the big…

36
r/LocalLLaMA community 13d ago

$5400 eBay 8x V100 server cranks on flash-next

Had opus 5.5 do the bring up.. super solid results. Using only 4 GPU results in KV of like 120k with image on. 27B TP=4 results in >200 tok/s with dflash, prefill around 2.5-3.5k. Both these are running the nvfp4 Nvidia checkpoints. Found a magic repo that unpacks into fp16 on…

22
r/LocalLLaMA community 13d ago

Why am I like this? (Full Chat and Image generation on a 286 Tandy 1000 TL/3)

40 year tech gap? No problem! The Tandy runs DeskMind, a native DOS program. It talks over WiFi (a PicoMEM 2 card with mTCP) to a small Python server on my PC. That server drives Qwen3.8-27B (NInfer on a 5090) and Krea 2 (ComfyUI on a 4090). The 286 never sees JSON, base64 or a…

28
r/LocalLLaMA community 13d ago

Qwen4Exp: add MTP by am17an · Pull Request #29761 · ggml-org/llama.cpp

now you can use MTP with Qwen Flash Next, time to switch from Qwen 3.8 27B? (merged after 17h of development) quants: https://huggingface.co/ggml-org/Qwen3.8-Flash-Next-GGUF link to the previous discussion (I deleted the old post to avoid duplicates):…

14
r/LocalLLaMA community 13d ago

Dual GPUs vLLM resources for 50 + 40 nvidia GPUs

i wanted to see how far i can push my PC and on the way i discoverd the resources are limited so i made my own and i hope it helps . https://github.com/ExTV/dual-gpus-vllm   submitted by   /u/Fz1zz [link]   [comments]

30
r/LocalLLaMA community 13d ago

12hrs with Ling 3.1 Flash

Made the hardest instrument to play with Ling 3.1 Flash 560B A25B (coming soon). It's kinda crazy that 'flash' is in the 500B category these days, but hey, it works. Can't wait to see what y'all will do when they release this model.   submitted by   /u/jaykayenn [link]…

20
r/LocalLLaMA community 13d ago

Best OpenSource Claude Cowork alternative?

Hey guys, Looking for an alternative for Claude Cowork : Project Work / Documents Integrations like Notion, Gmail, etc. Tools like Websearch, PDF creation, etc. Came over Eigent ( https://github.com/eigent-ai/eigent ) but cant find any actual reviews about it, what usually is a…

18
r/LocalLLaMA community 13d ago

DeepSeek harness 0.2 - Optional Bundle Architecture, Windows Sandbox improvements, Async Question Mode, Desktop release, Web Search without key

Optional Bundle architecture : Schedule (session-local delayed / timed / interval reminders) was removed from the default set and made an explicit Optional Bundle. This cleanly separates “installed” from “enabled” and is the first systematic use of the Profile + Bundle model for…

34
r/LocalLLaMA community 13d ago

Qwen Flash Next MTP work restarted

If you're using MTP with Qwen Flash Next and llama.cpp, you can switch to: quants: https://huggingface.co/ggml-org/Qwen3.8-Flash-Next-GGUF PR: https://github.com/ggml-org/llama.cpp/pull/29761 Please note that this is still wip   submitted by   /u/jacek2023 [link]  …

25
r/LocalLLaMA community 13d ago

Strata: how to configure sampling parameters

The top-level README doesn't mention this, but you can add a "sampling" key to your strata-iq3_s.json like this: { "sampling": {"temperature": 1.0, "top_p": 0.95, "top_k": 20}, "exe": "/path/to/Strata/engine/strata", "args": [ ... ], ... } From docs/DETAILS.md : The run config's…

25
r/LocalLLaMA community 13d ago

Finally got my 4th card in (64gb total)

I was waiting on the blowers for the T4s and posted this before it was finished, other than some braided cable sleeves for the fan wires its pretty much good, I was going to upgrade the CPU, but I'm getting great speeds comparatively to a CPU in my old box that had way more…

28
r/LocalLLaMA community 14d ago

Astrabox - Open source Arcade Game Generator

Hey everyone! I’ve been working on ASTRABOX : an arcade interface where you describe a game, the AI builds it, and you can ask for changes by voice while playing. Each game gets its own visuals and gameplay, while a shared runtime handles controllers, scores, player joining,…

8
r/LocalLLaMA community 14d ago

tp=6 can work on vLLM, with padding

vLLM's tensor parallel requires that several of the model architecture numbers be evenly divisible by the number of GPUs selected for tensor parallel. This usually means that you can only use a number of GPUs that is a power of two (2, 4, 8, etc.). I have six GPUs, and I want to…

8
r/LocalLLaMA community 14d ago

szmcp: a ZIM HTML to Markdown converter and yet another ZIM MCP server

Hello all, I wanted to share a little project I vibe-coded for myself that you may find useful. As many people here like to suggest, I wanted to give my small local LLMs access to information to improve their world knowledge. I didn't want to give my LLM free reign searching and…

24
r/LocalLLaMA community 14d ago

AI CEO Interviews (2026)

  submitted by   /u/LambdaHominem [link]   [comments]

10
r/LocalLLaMA community 14d ago

Least to most expensive (Somewhat modern) GPU's with 32gb of vram (Under $1600) Based on ebay listings

I was researching prices on ebay and fed claude a bunch of images of listings. I had it make a chart and thought it would be useful to share.   submitted by   /u/Rombodawg [link]   [comments]

16
r/LocalLLaMA community 14d ago

Best local model for Blender and game dev?

I have been looking at some local models, but even the smartest ones like GLM5.3 Flash and DSv4 Flash have a hard time creating coherent models in Blender and placing them logically in game engines. Is this something that local models are just too dumb still to do good job at?…

19
r/LocalLLaMA community 14d ago

Two open-weights releases: Victoria (Qwen3.8-Flash-Next with 44% of experts cut, 70% Terminal-Bench 2.1, GGUF included) and Maple (a Canada-first fine-tune)

We had a Dell B300 in the lab for a few weeks and used it to create two fine tunes of Qwen Flash Next. Victoria (coding and agents) Qwen3.8-Flash-Next cut down by 44% using a paper / technique called REAP: 512 down to 288 per layer. Retrained at 4-bit (NVFP4) afterwards, so it's…

21
r/LocalLLaMA community 14d ago

Open source inference engine (like LM Studio or Unsloth Desktop) that optimizes itself for your exact hardware. Compiles and tunes its kernels on your device, so open models run up to 2x faster than llama.cpp. Works on Apple Silicon, NVIDIA, AMD or nothing but a CPU.

  submitted by   /u/paranoidray [link]   [comments]

34
r/LocalLLaMA community 14d ago

peculiar-ragdoll's Dirk-Qwen 3.8-27B vs. UkisAI Swift-1.5 Qwen3.8-27B

I have a long list of my own domain specific eval questions that I run to validate which models I can rely on: coding, coding (numpy/pandas), data analytics decision making, local RAG, and voice assistant. It's made up of the types of things I'm likely to deal with on the daily.…

6
r/LocalLLaMA community 14d ago

If one hour of AI is costing me 0.12€ is paying for frontier a cheaper option?

Running Qwen flash next of even Qwen 27b dense, I can do any,burning sticking to flash due to its speed, and the kwh cost is at 0.25€ where I live in, ranging from 0.11€ to 0.35€, so I used chatgpt to help me calculate the total kwh consumption on my 7900xtx plus 9800x3D, and…

26

Two 96 GB Ascend cards crun Qwen3.8-flash-next hardware notes, vLLM work, benchmarks, and what is next

Inference Engineering for Dummies

I made my own cybersecurity benchmark and ran Qwen3.8 27B, here's how a local model actually does at hacking

Apex Compute developing open-source Mesa Vulkan driver for their hardware

Qwen3.8-Flash-Next (5.05bpw + ngram at bf16) exl3 on one r9700: 863 t/s prefill and 35 t/s decode at 230k context (256k max), is that ok or am i missing something?

llama, server: add /v1/systemone API (models: laya, julia-1, lev, openjev, kev) by ngxson · Pull Request #29818 · ggml-org/llama.cpp

Does anyone know if any new releases from Mistral are planned?

best <40B alternatives to Qwen/Deepseek for (1) Coding (2) Long document QA test

Mirostat?

Detecting hallucinations in local models without eating VRAM: What we learned testing 1.5B to 120B models

Update on the “Monstrosity”. 6 BC-250 board cluster

Introducing ModelHarbor, a self-hosted model library

I "built" a local LLM radio like the Pharaohs built the pyramids

Perplexity Decider 27B: Open weights decision model fine tune of Qwen3.8 27B

Pi 1.0 released - MCP support now included by default

Gufo performance .... 70tps Qwen 3.8 27b but you need to read the fine print.

What local AI model is good for game decomps/recomps?

DDR4/PCIe4 vs DDR5/PCIe5 for LLMs- I benchmarked them for pre-training. What are your thoughts?

Running 95.5 GiB Qwen3.8-Flash-Next at 41–52 tok/s on a 64GB Mac (1.76x faster than llama.cpp): Slipstream release, 130k context scaling, + Swift variant

AMA about K2 Horizon, Meet our team from IFM

Now which lab is behind fledge alpha?

What's the best setup for Qwen3.8 27b for a 16 gig VRAM?

Update #2: Post training yandex/AliceAI-80B-A3B [instruct!] from scratch

Clef: Open Weights decision model by Cloudflare

Heretic is on PewDiePie!

Unsloth, Swift1.5, Peculiar-Ragdoll, ThinkingCap - Qwen3.8-27B

Pi extension: Skip reasoning with local Qwen 27B and proceed to answer right now

We benchmarked 18 RAG pipelines against an agent loop on Google's FRAMES. The best pipeline hit 78.9%. The agent loop hit 92.7%.

China and the memory market

Jeff-Qwen3.5-0.8B v1.2 + 9 LoRA adapters: put it in front of Qwen3.8-27B for 38× faster decisions and +8.7 points accuracy, for under 2 GB extra memory

$5400 eBay 8x V100 server cranks on flash-next

Why am I like this? (Full Chat and Image generation on a 286 Tandy 1000 TL/3)

Qwen4Exp: add MTP by am17an · Pull Request #29761 · ggml-org/llama.cpp

Dual GPUs vLLM resources for 50 + 40 nvidia GPUs

12hrs with Ling 3.1 Flash

Best OpenSource Claude Cowork alternative?

DeepSeek harness 0.2 - Optional Bundle Architecture, Windows Sandbox improvements, Async Question Mode, Desktop release, Web Search without key

Qwen Flash Next MTP work restarted

Strata: how to configure sampling parameters

Finally got my 4th card in (64gb total)

Astrabox - Open source Arcade Game Generator

tp=6 can work on vLLM, with padding

szmcp: a ZIM HTML to Markdown converter and yet another ZIM MCP server

AI CEO Interviews (2026)

Least to most expensive (Somewhat modern) GPU's with 32gb of vram (Under $1600) Based on ebay listings

Best local model for Blender and game dev?

Two open-weights releases: Victoria (Qwen3.8-Flash-Next with 44% of experts cut, 70% Terminal-Bench 2.1, GGUF included) and Maple (a Canada-first fine-tune)

Open source inference engine (like LM Studio or Unsloth Desktop) that optimizes itself for your exact hardware. Compiles and tunes its kernels on your device, so open models run up to 2x faster than llama.cpp. Works on Apple Silicon, NVIDIA, AMD or nothing but a CPU.

peculiar-ragdoll's Dirk-Qwen 3.8-27B vs. UkisAI Swift-1.5 Qwen3.8-27B

If one hour of AI is costing me 0.12€ is paying for frontier a cheaper option?