r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 2d ago

Bolt Graphics GPU will have 2 DDR5 laptop DIMM slots

They have a few working prototypes, & are aiming for pre-production examples made by end of this year, & full production by Christmas 2027. Interesting specs: 5nm GPU "High performance CPU in GPU" on-card LPDDR5X as primary memory pool 2 DDR5 SODIMM slots for 'spill over'…

38
r/LocalLLaMA community 3d ago

Anyone else end up building a web access layer for local AI agents?

I've been running local models for most of my experiments, and I kept running into the same issue. The model lives locally, but everything it needs to interact with doesn't. Every new agent ended up with another GitHub client, another Reddit integration, another documentation…

10
r/LocalLLaMA community 3d ago

Mellum2 local deployments

Hey local community, I work at JetBrains with the team that trained Mellum2 models — 12B-2.5A LLMs. Those models are trained completely from scratch, targeting fast inference: our primary goal were H100/H200s prod deployments, but local deployments are good as well. We…

37
r/LocalLLaMA community 3d ago

NASA testing local LLM inference for future space missions

Red Hat published a blog post last week about an initiative I supported with NASA researchers at Johnson Space Center building a medical AI assistant. It's called the Crew Medical Officer Digital Assistant (CMO-DA) and the system runs LLMs and other models on local hardware with…

34
r/LocalLLaMA community 3d ago

Kimi and GLM on frontier code

  submitted by   /u/Charuru [link]   [comments]

36
r/LocalLLaMA community 3d ago

CPU-only GLM 5.2: Epyc and 512GB RAM

This is just a preview of some content I'm putting together to share with you all. I have a server I've put together and I'm testing the 4-bit version of GLM 5.2 (GLM-5.2-UD-Q4_K_XL). This is an Epyc Rome 7452 with 512GB of RAM. TLDR: This is the unedited prompt, response and…

28
r/LocalLLaMA community 3d ago

Any good uses for a 192 GB DDR3 Server in the LLM world?

I've been gifted this old IBM System X V4 with a dual Xeon E5-2640 [6c12t @ 2.7 GHz] and a whooping 192 GB of DDR3 1666 ECC RAM There's a gen 2 x16 PCi-E port in there as well so it can take a single GPU... Does anyone have some fun ideas on what to do with this system? It's…

15
r/LocalLLaMA community 3d ago

Is it ever possible to have a malicious LLM with a backdoor

I was just brainstorming of possibilities that the LLMs behave differently than normal if trained to recognize a specific secret sentence , and then unlocks a backdoor of malicious behavior. This sounds to me very possible at first glance. Don't get me wrong, the risk is…

32
r/LocalLLaMA community 3d ago

Deepseek V4 Official Launch to be released mid-July with API price changes

Is this the official release for deepseek? I hope it has huge improvements https://preview.redd.it/dm5l0qn8k7ah1.png?width=694&format=png&auto=webp&s=12eadfd0a52c0f1a65bcd685f2cdbb29aff457be   submitted by   /u/jmorant555 [link]   [comments]

22
r/LocalLLaMA community 3d ago

Apparently you can skip entire transformer blocks at load time with minimal performance impact

The benefit is another trick to allow fitting a model that wouldn’t fit in your hardware otherwise. People currently rely on quantization, and this is just another tool that can be used for that purpose (and they can be used together as well) Following recent (very cool) papers,…

30
r/LocalLLaMA community 3d ago

DeepSeek V4 official version will be launch on mid-July

https://preview.redd.it/n7rwh262b7ah1.jpg?width=1024&format=pjpg&auto=webp&s=33d775b456843cd2dbd458de89384a6a7d6d87d1 Source: Email sent from deepseek (email only available for chinese user) used gpt image 2 translate image into english   submitted by  …

34
r/LocalLLaMA community 3d ago

DeepSeek V4 by am17an · Pull Request #24162 · ggml-org/llama.cpp

now you can run DeepSeek V4 locally   submitted by   /u/jacek2023 [link]   [comments]

26
r/LocalLLaMA community 3d ago

GLM 5.2 Q1_S vs Qwen 27B Q8

TL;DR; GLM-5.2 Q1_S beats Qwen 3.6 27B Q8, both run at KV Q8 edit: GLM run a K & V Q8, Qwen run with KV cache at full FP16., with preserve thinking on. Disclaimer : This is a hobby/amateur comparison with n=1, so go easy on it. I just thought it would be fun to share. The…

11
r/LocalLLaMA community 3d ago

LibreChat or OpenWebUI ?

Hello, I have a friend that while technical, it doesn't know too much about AI, I've helped them with the infrastructure setting and that works like a charm, but he's interested in a thing that where I don't have too much experience with, and that is flashy chat "do everything"…

14
r/LocalLLaMA community 3d ago

MiCA is now part of Hugging Face PEFT

Glad to share that MiCA, short for Minor Component Adaptation, has now been merged into the HuggingFace PEFT library. It is not yet included in the latest PyPI release, but you can already install it directly from PEFT main: pip install --upgrade…

18
r/LocalLLaMA community 3d ago

AMD MI210 64GB vs DCU K100 64GB

On the Chinese eBay there is a many DCU K100 64 GB GPU available for a very attractive price, between 6000 RMB and 19 000 (air or water cooled versions, new or second hands), and 15 000 to 20 000 for the AMD MI210 (4000-6000 RMB for the PCIE bridge). There is very little…

25
r/LocalLLaMA community 3d ago

Update: First Manual Results from Testing Procedural Skill Transfer in Small Models

Yesterday I posted an idea for testing whether a large model can transfer some of its procedural skill to a smaller model without fine-tuning. The short version of the idea was this: Small models are often not completely lacking knowledge. They know the syntax. They know the…

18
r/LocalLLaMA community 3d ago

Minimax M3 vs M2.7

M3 has been out for ~2 weeks now. Would love to hear feedback from those who have updated to M3 from M2.7.   submitted by   /u/rm-rf-rm [link]   [comments]

32
r/LocalLLaMA community 3d ago

High-quality GLM-5.2 Quant on 4x DGX Spark - Guide, Results, and Comps

I got GLM-5.2 NVFP4 running on four DGX Sparks at 128K context. This is still a niche/hacky setup, but it is now a real serving point rather than just a proof of life. Objective : A high quality 4-bit quant running on 4x spark. Model: https://huggingface.co/Mapika/GLM-5.2-NVFP4…

9
r/LocalLLaMA community 3d ago

MLX Fine-Tune Example Guide

A Local MLX Fine-Tuning Experiment Just finished a local LoRA fine-tune of a 7B instruction model on Apple Silicon, via MLX, teaching it a high-fantasy literary register (Gene Wolfe and Tolkien). This is a more rigorous version with more data of something I tried two years ago…

14
r/LocalLLaMA community 3d ago

I built an agent Harness for Small Models. I got Qwen 3.5 4b managing servers.

This is something I've been working on, I like playing around with smaller local models but found most agent harness's not well suited for them. The failure modes across different model family's tend to be the same: Failed tool calls Poor varication of environment variables Poor…

12
r/LocalLLaMA community 3d ago

Locally running mode turns an Image into a Cute Controllable Character you can Play as

This is a sequel to my last post here !! It meant a lot to have such positive feedback last time. This is the 800M version of the previous model. It still has a LOT of issues but the promise is the same. Working comfortably on consumer GPUs The context is increased to 12 latent…

32
r/LocalLLaMA community 3d ago

Qwen3.6-27B UD Q3 with kv at q8 is quite amazing for simple proof of concepts

Preface, technology is not my industry, but I am a very passionate poor man. So much so that I discovered 'AI' - ChatGPT in the beginning of 2025. So go easy on me, I only try. I kind of understand MOE vs. Dense models, MOEs are much forgiving when it comes to running as there…

22
r/LocalLLaMA community 3d ago

NPC Engine Using Local Models

I’ve been working on a game-agnostic NPC engine/backend based pretty heavily on SillyTavern-style architecture, and with smaller local models getting better and better, I honestly think this kind of thing could be the future of RPGs. Right now I’m using NVIDIA Parakeet 0.6 for…

22
r/LocalLLaMA community 3d ago

Tensor split performance on low-bandwidth (TB3) eGPUs, and a question

Hey everyone! I've got a pair of Morefine G1 4090M 16gb eGPUs connected at 40Gbps via TB3 (daisy-chained). I normally run them in layer split mode as it doesn't seem to need much bandwidth; I'm seeing around 1300t/s PP and 26t/s TG (35-40 with MTP), qwen3.6-27B @ Q4. Which is…

20
r/LocalLLaMA community 3d ago

Trying to understand why so many trash fine-tuned models on HuggingFace ...

The majority of these models do not perform even as well as the base model, not even worth wasting the disk space on HuggingFace server, Qwhoppass-27B-Mother-Ultimate-Lord, whatever... Seeing their proliferation and the booming AI job market, I think many of those are just for…

14
r/LocalLLaMA community 3d ago

Success story with MiMo-V2.5-GGUF:UD-Q5_K_XL

I don't see many stories about this model, but after several attempts (after I finished finally reconfiguring my cluster) I did something useful with it: it wrote a built-in llama.cpp tool for executing C++ code and using the results. Here's an exercise that MiMo V2.5 gave me to…

27
r/LocalLLaMA community 3d ago

Ornith-1.0-35B GGUF update: native MTP speculative-decode graft + full serving/TTFT/long-context numbers (llama.cpp, tp=1)

Follow-up to my previous Ornith-1.0-35B Q3_K_M post. I grafted a native MTP draft head onto the IQ4_XS body (head at Q6) for self-speculative decode, single GPU, llama.cpp: 1.3-1.35x single-stream decode (172.6 -> 233.8 tok/s). Next-token distribution is byte-identical to…

11
r/LocalLLaMA community 3d ago

China Has Matched Anthropic in Cybersecurity, Resetting AI Race

  submitted by   /u/pscoutou [link]   [comments]

30
r/LocalLLaMA community 3d ago

A lot of good M5 Max options available at Apple Refurbished

Just a heads-up. After Apple's price hike announcement, they added a bunch of top-of-the-line 14" M5 Pro/Max options to their refurbished website. If you got discouraged by the price hike, check out their refurbished store.   submitted by   /u/Hanthunius [link]  …

13
r/LocalLLaMA community 3d ago

The number 1 public enemy of open-source.

Dario's args: "Opensource you can see the source, here you cannot see inside the model" - yes you can that's literally the open weights part btw. - I cannot see the weights inside Claude, but I can GLM 5.2 - Models like Nemotron3 Ultra go further, all the data, training scripts,…

25
r/LocalLLaMA community 3d ago

Script to monitor llama cpp and analyze memory usage

My goal has always been to be productive with commodity hardware. So far my workhorses have been the MoE editions of gemma 4 and Qwen 3.6 on an old desktop with a single 9060XT with 16GB ram. The problem has always been that every source is vague about Vram/ram requirements.…

33
r/LocalLLaMA community 3d ago

Hypothetically speaking...

Would it not be possible to create crowd sourced, truly open sourced distilled LLMs with a simple wrapper around command line based AI services that exist today? I'm imagining a layer that goes around whatever application people currently use for coding/AI boyfriend that…

24
r/LocalLLaMA community 4d ago

DeepSpec - a deepseek-ai Collection

DeepSpec DeepSpec is a full-stack codebase for training and evaluating draft models for speculative decoding. It contains data preparation utilities, draft model implementations, training code, and evaluation scripts. Released Checkpoints The checkpoints below are the ones used…

26
r/LocalLLaMA community 4d ago

DFlash support merged into llama.cpp

  submitted by   /u/sammcj [link]   [comments]

36
r/LocalLLaMA community 4d ago

Sell ddr5 for vram?

Hi, I have 768gb ddr5 6400 ecc ram, given current ram prices should I sell half and buy rtx 6000 pros? Edit: currently have an epyc 9255 and 2 x rtx 6000 pro max q but could add another three Blackwell potentially then and run something like glm 5.2 in q4 at faster speeds then…

6
r/LocalLLaMA community 4d ago

Qwen3.6 27B local vs Opus 4.8, voxel engine in raw C with zero frameworks

Sunday experiment. Same prompt to both. Build a voxel world in plain C. No engine, no game library, no framework, just the compiler. The model does its own chunk meshing, render loop and memory management by hand. Left is Claude Code on Opus 4.8. Right is Qwen3.6 27B local on…

37
r/LocalLLaMA community 4d ago

Best case for dual RTX 3090 (250W each) on Crosshair VIII Hero?

I'm building a local LLM workstation and would appreciate some advice from people already running 2×3090s. Current hardware: ASUS Crosshair VIII Hero (X570) One Gainward Phoenix RTX 3090 Looking for a second used 3090 (not necessarily the same model) Both GPUs will be…

9
r/LocalLLaMA community 4d ago

Are there good closed vs open LLM rankings? Also, are 70B–350B models actually worth it?

hey, I’m currently getting enough VRAM to run something in the GLM-5.2 range, but I’m wondering: do we actually have a solid ranking that compares closed-source and open-weight LLMs side by side? I’ve been trying to find a clear “closed vs open” leaderboard, but most benchmarks…

26
r/LocalLLaMA community 4d ago

How many of you do use Q1 or Q2 of Big models(100-250B)? How's it?

Sharing popular(also recent) models for reference: 151-250B : DeepSeek-V4-Flash Step-3.X-Flash Command-a-plus-05-2026 Laguna-M.1 MiniMax-M2.X Qwen3-235B-A22B 100-150B : GLM-4.5-Air Qwen3.5-122B-A10B NVIDIA-Nemotron-3-Super-120B-A12B Mistral-Small-4-119B-2603…

34
r/LocalLLaMA community 4d ago

"DuckDuckGo is blocking with a CAPTCHA. Let me try other approaches:"

My local llama.cpp-based LLM just started reporting this this morning: "DuckDuckGo is blocking with a CAPTCHA. Let me try other approaches:" Is anyone else seeing this with DuckDuckGo?   submitted by   /u/SensitiveCranberry00 [link]   [comments]

34
r/LocalLLaMA community 4d ago

This application to join the GPT 5.6 Sol preview is wild

+ face scanner, fingerprint checker and passport swiper   submitted by   /u/Complete-Sea6655 [link]   [comments]

16
r/LocalLLaMA community 4d ago

A barebones CPU-only inference engine for Qwen 3, written from scratch in pure C

TL;DR: The (very messy) code and writeups can be found at https://github.com/jakint0sh/qwen3-engine Read the README for instructions on how to get started. And for those who just want a bulleted list: - Inference engine for Qwen 3 sizes 4B and below - Written from scratch in…

37
r/LocalLLaMA community 4d ago

We're probably going to need that soon.

From: Vladik on 𝕏: https://x.com/Kostoglodov/status/2071144065857679631 Shaw (spirit/acc) on 𝕏: https://x.com/shawmakesmagic/status/2070918006033817867   submitted by   /u/Nunki08 [link]   [comments]

8
r/LocalLLaMA community 4d ago

Whisperian: It is one of the best applications for Android, if you want to use Mic with some local ASR models. And it is also available on Play Store.

  submitted by   /u/9r4n4y [link]   [comments]

29
r/LocalLLaMA community 4d ago

Is Qwen3-VL-2B the only viable VLM for JSON extraction on a "potato"?

After spending countless hours testing on 3 "potato" laptops (Intel i3, 8GB RAM, Win11, integrated GPU), that's my conclusion. For reliably extracting data from images to JSON on low-end hardware, nothing else even comes close. Yet, it’s completely missing from major benchmarks…

23
r/LocalLLaMA community 4d ago

clark-labs/clark-air-sana-1.6b-1.58bit · Hugging Face

A Sana 1.6B text-to-image transformer compressed to ternary (~1.85 bits/weight): 8.6× smaller than FP16, near-FP16 quality. Footprint (measured) Artifact Size vs FP16 What it is FP16 transformer 3.21 GB 1× (100%) reference Clark Air (packed) 374 MB 8.6× (≈12%) packed ternary (…

36
r/LocalLLaMA community 4d ago

A Blind Visual Paradigm for Testing Skill Transfer in Small Models Without Fine-Tuning

TL;DR: Small models aren't dumb, they're shallow. I designed a cross-domain, blind, visual experiment to see if a large model can compress its "planning discipline" into a reusable scaffold that makes a small model deeper — with zero fine-tuning. Three.js is the testbed because…

28
r/LocalLLaMA community 4d ago

Finally.. my rig is maxed out

Got all the parts before the crazy price increase except for the rtx pro 5k! Was saving up to order rtx pro 6000 in US and i did, but wanted to join nvidia inception program for the discount. It was around $8.5k during that time, less 1k if I succeeded. It took around 3 months…

7
r/LocalLLaMA community 4d ago

US Ban Benchmark Updated: Toe-to-toe Between Two Big Names!

OpenAI ties with Anthropic in this benchmark following the preview of GPT 5.6 just yesterday. Chinese models have no hope of catching up forever, while Gemini's figure is yet to be updated.   submitted by   /u/Complete-Sea6655 [link]   [comments]

30

Bolt Graphics GPU will have 2 DDR5 laptop DIMM slots

Anyone else end up building a web access layer for local AI agents?

Mellum2 local deployments

NASA testing local LLM inference for future space missions

Kimi and GLM on frontier code

CPU-only GLM 5.2: Epyc and 512GB RAM

Any good uses for a 192 GB DDR3 Server in the LLM world?

Is it ever possible to have a malicious LLM with a backdoor

Deepseek V4 Official Launch to be released mid-July with API price changes

Apparently you can skip entire transformer blocks at load time with minimal performance impact

DeepSeek V4 official version will be launch on mid-July

DeepSeek V4 by am17an · Pull Request #24162 · ggml-org/llama.cpp

GLM 5.2 Q1_S vs Qwen 27B Q8

LibreChat or OpenWebUI ?

MiCA is now part of Hugging Face PEFT

AMD MI210 64GB vs DCU K100 64GB

Update: First Manual Results from Testing Procedural Skill Transfer in Small Models

Minimax M3 vs M2.7

High-quality GLM-5.2 Quant on 4x DGX Spark - Guide, Results, and Comps

MLX Fine-Tune Example Guide

I built an agent Harness for Small Models. I got Qwen 3.5 4b managing servers.

Locally running mode turns an Image into a Cute Controllable Character you can Play as

Qwen3.6-27B UD Q3 with kv at q8 is quite amazing for simple proof of concepts

NPC Engine Using Local Models

Tensor split performance on low-bandwidth (TB3) eGPUs, and a question

Trying to understand why so many trash fine-tuned models on HuggingFace ...

Success story with MiMo-V2.5-GGUF:UD-Q5_K_XL

Ornith-1.0-35B GGUF update: native MTP speculative-decode graft + full serving/TTFT/long-context numbers (llama.cpp, tp=1)

China Has Matched Anthropic in Cybersecurity, Resetting AI Race

A lot of good M5 Max options available at Apple Refurbished

The number 1 public enemy of open-source.

Script to monitor llama cpp and analyze memory usage

Hypothetically speaking...

DeepSpec - a deepseek-ai Collection

DFlash support merged into llama.cpp

Sell ddr5 for vram?

Qwen3.6 27B local vs Opus 4.8, voxel engine in raw C with zero frameworks

Best case for dual RTX 3090 (250W each) on Crosshair VIII Hero?

Are there good closed vs open LLM rankings? Also, are 70B–350B models actually worth it?

How many of you do use Q1 or Q2 of Big models(100-250B)? How's it?

"DuckDuckGo is blocking with a CAPTCHA. Let me try other approaches:"

This application to join the GPT 5.6 Sol preview is wild

A barebones CPU-only inference engine for Qwen 3, written from scratch in pure C

We're probably going to need that soon.

Whisperian: It is one of the best applications for Android, if you want to use Mic with some local ASR models. And it is also available on Play Store.

Is Qwen3-VL-2B the only viable VLM for JSON extraction on a "potato"?

clark-labs/clark-air-sana-1.6b-1.58bit · Hugging Face

A Blind Visual Paradigm for Testing Skill Transfer in Small Models Without Fine-Tuning

Finally.. my rig is maxed out

US Ban Benchmark Updated: Toe-to-toe Between Two Big Names!