r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 7d ago

Tested in Coding: Strata

This will be a potentially unpopular post - but it's the truth and grounded - so let's get to it. Hopefully you are familiar with my previous Tested in Coding series: https://www.reddit.com/r/LocalLLaMA/comments/1vvsokm/tested_in_coding_q8_k_xl_qwen38_27b_vs_bf16/…

5
r/LocalLLaMA community 7d ago

Surface RTX Spark Dev Box: The Dev Box Built For Developers

$5995 - Ships in November. N1X brand of GB10 Chip. Says it will have WSL out the door which I presume will be Ubuntu + Cuda beneath in addition to all the CoPilot/GitHub native stuff for Windows. Hopefully they have supply to saturate the market and put in pricing pressure.…

17
r/LocalLLaMA community 7d ago

[R] Split the Differences, Pool the Rest: Provably Efficient Multi-Objective Imitation

TLDR: The question we answer: how do you learn from experts with different objectives? Pooling all their data can lose their trade-offs; learning from each expert separately misses opportunities to share data. MA-BC pools demonstrations where observed actions don’t disagree,…

11
r/LocalLLaMA community 7d ago

Qwen 3.8 Flash-Next on 64GB RAM + 16GB VRAM, worth it over a fully-loaded 3.6 35B-A3B?

4080 16GB, 64GB RAM. Currently on 3.6 35B-A3B (Unsloth quant, fits fully in VRAM) and happy with it. Anyone tried Flash-Next at Q2/Q3 on a similar setup? Worth the switch, or should I stay put? Asking before committing to a 30GB+ download.   submitted by   /u/msalsas…

14
r/LocalLLaMA community 7d ago

llama.cpp on the stage

https://x.com/ggerganov/status/2107908265032229261   submitted by   /u/jacek2023 [link]   [comments]

26
r/LocalLLaMA community 7d ago

Kandinsky 6.0 video gen + video upscaler!

Kandinsky 6.0 Pro (29B) and Lite (3B). Already support for ComfyUI and Diffusers. https://github.com/kandinskylab/kandinsky-6   submitted by   /u/KokaOP [link]   [comments]

34
r/LocalLLaMA community 7d ago

llama : add a GPU cache for MoE experts kept in host memory by am17an · Pull Request #29887 · ggml-org/llama.cpp

Potentially big speedup for MoE models that don’t fully fit in VRAM. Are you GPU Poor? Show your speedups ;)   submitted by   /u/jacek2023 [link]   [comments]

37
r/LocalLLaMA community 7d ago

omni-d1 600M by Liquid AI running in the browser using WebGPU

Liquid just dropped d1-omni-600M and it's a decision model! I ported it to runntime, the WebGPU inference library I'm currently working on. It's plain TypeScript on top of TypeGPU with no WASM and virtually no export step. The model is written directly from our core ops (matmul,…

34
r/LocalLLaMA community 7d ago

d1-3B and d1-omni from LiquidAI

https://preview.redd.it/owhrvbbiu2uh1.png?width=4096&format=png&auto=webp&s=55d816753e7adee0e1391c9f581052b4bde44da7 d1-omni-600M d1-omni-600M is a 600M parameter decision model built on LFM2.5-Encoder-350M . You give it a state (text or JSON, with images or a voice clip) and a…

28
r/LocalLLaMA community 7d ago

LiquidAI/d1-3B · Hugging Face

  submitted by   /u/iamn0 [link]   [comments]

33
r/LocalLLaMA community 7d ago

A week in Beijing and Shanghai with the people building AI in China

I thought this was a very interesting article, of relevance to the readers here.   submitted by   /u/uutnt [link]   [comments]

9
r/LocalLLaMA community 7d ago

GPU Upgrade advice

Upgrade Question If we say the "budget max" is $1700-1800, and this could include upgrading to a Taichi motherboard: With a new motherboard , no need to bifurcate 1. Would you add a second 5060Ti (16GB)? Someone is selling one for around $500 locally; 2. Buy a used 7900 XTX…

35
r/LocalLLaMA community 7d ago

Local Qwen 3.8 27B vs DeepSeek Flash API: Is local good enough?

Running a model on your own machine used to be a privacy story with a quality tax. On this generation that trade has narrowed to where we can state it plainly: for daily work, the local model is good enough. We measured it — 25 paired tasks across four workloads, same prompts,…

36
r/LocalLLaMA community 7d ago

feat: add GLM5Next MTP, optimize by pwilkin · Pull Request #29928 · ggml-org/llama.cpp

now you can use GLM 5 Flash MTP locally   submitted by   /u/jacek2023 [link]   [comments]

12
r/LocalLLaMA community 7d ago

New LFM to be released today

https://x.com/ramin_m_h/status/2107780594264600801 What size do you want? https://huggingface.co/LiquidAI   submitted by   /u/jacek2023 [link]   [comments]

30
r/LocalLLaMA community 7d ago

Seeking upgrade advice

I got dual 7900xtx running on a z390 (pcie3 8x) running the 27b. I've been thinking upgrading the motherboard to a x570 (pcie4 8x) or a x870 (pcie5 8x) improving performance with TP, and then wait to buy medusa or spark with lpddr6 and 256GB (apple isn't an option for me).…

32
r/LocalLLaMA community 7d ago

Micron Says NVHBM to Improve Profitability Even With Outsourced Base Die

"NVHBM moves the memory controller, which was previously located on the main compute die, into the base die. This reduces power consumption by 15% and increases memory bandwidth by as much as 30%. It also integrates a customized physical layer (PHY) for input/output (I/O),…

22
r/LocalLLaMA community 7d ago

cmpunlocker v0.5 just dropped, ECC support along with 4 extra SM unlocked for FREE, who needs a 64GB DGX Spark when you've got a CMP170hx right? 1.5TB/s memory BW vs 273 GB/s, all for less than 1/2 the price of a 64GB DGX Spark

If you're like me and saw the price increase for the 128gb DGX Spark go from 4.7k -> 7k while a new version with 64GB launch for 5k, you'll have been very disappointed and every right to be so, it's just plain sad for localAI. Well, here's some good news, cmpunlocker v0.5 just…

6
r/LocalLLaMA community 7d ago

Image-text retrieval with EmbeddingGemma 2's vision tower, running in the browser on WebGPU

EmbeddingGemma 2 came out this week. It maps images and text into one 768-dim space, so you can search photos by describing them. I ported its text and vision towers to ruNNtime, a WebGPU inference library in TypeScript, and made a small photo gallery where search runs entirely…

28
r/LocalLLaMA community 7d ago

A stock GLM-Edge-1.5B-Chat on a 4GB Galaxy A04e completed a real Amazon cart task

Yesterday TechCrunch published a piece about a growing problem for AI agents: websites are starting to block them. Amazon blocking Meta's Muse is the obvious example. At almost exactly the same time, GLM-Edge-1.5B-Chat running locally on a 4GB Galaxy A04e completed a real Amazon…

22
r/LocalLLaMA community 7d ago

I re-trained the DFlash 2 drafter for Ternary Bonsai 2 27B: 2.2x on an L4 (3.2x on code edits with ngram lookup), 1.5x on a Mac, 1.2x in Chrome

PrismML's Ternary Bonsai 2 27B fits a 24 GB card or Mac, but decodes at ~30 tok/s on an L4 and ~21 on an M4 Pro. z-lab's DFlash 2 drafter was trained on bf16 Qwen3.8-27B, so it guesses worse on the ternary model. I fine-tuned it on 1.5M tokens of Bonsai 2's own greedy output.…

22
r/LocalLLaMA community 7d ago

Ramjet - mini altermative to nvidia dynamo

Hello, if you are running multi gpu setup, check out ramjet https://github.com/helixml/ramjet The goal is to have a local version of dynamo that can do equal or better job (but ideally without k8s). Check out https://helix.ml/blog/ramjet-vs-nvidia-dynamo as well. If you have dgx…

4
r/LocalLLaMA community 7d ago

Single 3090 Qwen 27B user, considering buying 128GB of RAM because of the hype

My current setup: - single 3090 running turboderp/Qwen3.8-27B-exl3:SC_5.00bpw_H6_V6 - ~150k context, ~70t/s, unknown prefill because I didn't benchmark it (but it is ok) - Intel 12400 CPU with 32GB DDR4 RAM All the hype around strata makes me consider buying 128GB of 6000MHz…

26
r/LocalLLaMA community 7d ago

New 35B-A3B model is comming on huggingface

MiniCPM-V-4.7-35B-A3B No model card yet   submitted by   /u/BreakfastFriendly728 [link]   [comments]

22
r/LocalLLaMA community 7d ago

Trained a ~20K LM (probably smallest) that can still write stories

I’ve been pushing TinyStories-style models downward in size, and this is the smallest one so far: MacroStories — 19,969 parameters, 81 KB FP32 https://huggingface.co/raincandy-u/MacroStories For scale: → ~50× smaller than the 1M TinyStories model → ~3,000× smaller than AlexNet →…

37
r/LocalLLaMA community 7d ago

Local RAG for a personal second brain: embedder and hybrid retrieval picks in 2026?

Building a fully local RAG setup for personal notes (life-logging second brain, single user, privacy is the whole point so no hosted APIs for the data). Stack is SQLite + sqlite-vec + FTS5, Ollama for embeddings and generation, all on an Apple Silicon Mac. Two questions where…

23
r/LocalLLaMA community 7d ago

How far we’ve come

  submitted by   /u/like_a_tensor [link]   [comments]

25
r/LocalLLaMA community 7d ago

BenchLabs' Cagliostro V3.5 135M finally takes Hugging Face's SmolLM2-135M's first place on Open SLM Leaderboard with an Intelligence Index of 27.49

  submitted by   /u/Megneous [link]   [comments]

9
r/LocalLLaMA community 7d ago

Why is ik_llama.cpp said to be faster than Mainline? On my hybrid multi-GPU rig, Mainline easily beats it

I constantly see recommendations saying that ik_llama.cpp (ikawrakow's fork) is the undisputed king of hybrid CPU/GPU offloading and MoE performance. However, every time I benchmark it against mainline ggml-org, mainline consistently beats it by a wide margin. Am I missing…

28
r/LocalLLaMA community 7d ago

Ugh I didn't want to post this... Back to Qwen3.8 27B

I don't know if anyone else has ran into these issues but when using Qwen Flash Next, my confidence in it at iq4_xs is high but not 100%. I notice it not following instructions, hallucinating more often and surprisingly it uses way less tokens than 27b. After using Strata....…

22
r/LocalLLaMA community 7d ago

Ruach Studio: a whole song studio around YuE2 on your own GPU. Score first, LoRA training, stems,remaster, DAW export.

We spent the last weeks building a studio around YuE2 , the open song model by m-a-p, and today it reaches its first release candidate. Write the style and the lyrics, and it composes, sings and renders the song on your own card. Nothing leaves your machine unless you point the…

23
r/LocalLLaMA community 7d ago

Are Tesla K80s any good for inference?

I'm seeing these show up on eBay for $50-$60. wouldn't expect anything ground breaking from it, but at that price, it seems like it could be interesting to play with on an extra pcie port. just curious if anyone's already done that   submitted by   /u/cmdr-William-Riker…

35
r/LocalLLaMA community 7d ago

MiMo V2.6 Pro almost matched GPT-6 Astra at figuring out what a user actually wants

Introducing InferBench: A benchmark testing how well frontier LLMs infer a user's priorities from their instructions. We tested 12 LLMs across 20 scenarios with 2.8k conversations to see which models understand the user's goals. Each conversation has a simulated user with a…

10
r/LocalLLaMA community 8d ago

I have about $4000, what's the best setup to get?

Ideally I'd like to be able to run Qwen 3.8-Flash-Next with decent performance. I was thinking of just buying 2x Radeon AI R9700 (64GB VRAM), or maybe a DGX Spark but that was before the price hike. My brother suggested just getting a Strix Halo box with 128GB unified. I did see…

19
r/LocalLLaMA community 8d ago

Qwen3.8-27B with just 12GB VRAM + 8GB RAM - ~18 tok/sec

I got Qwen3.8-27B running at ~18 tok/sec decode & ~500-600 tok/sec prefill (at 64k context) on just 12GB VRAM + 8GB RAM, using ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S quant (~11GB). This recipe also works with 0bserverx’ Qwen3.8-27B-Heretic-GSQ-RCO IQ3_S quant, achieving similar…

29
r/LocalLLaMA community 8d ago

pi-optchat: never compact again - endless chat as a memory tree

I built a Pi extension that implements Victor Taelin's OptChat recipe: instead of compacting, every message is logged and summarized into a binary tree. Each turn starts from a fresh context with a bounded memory view (128 KB), and the agent uses zoom/date to read the originals…

23
r/LocalLLaMA community 8d ago

Europe rejoins the fight with Chonky! Mistral Large 4 Released, Open weights end of month, who’s ready?

1 trillion parameters, 49B active, definitely chonky! If you don’t love the model you gotta at least love the humor in the name - Le Chonk   submitted by   /u/chemist_slime [link]   [comments]

13
r/LocalLLaMA community 8d ago

How abliterated models can get you pwned

Be safe out there boys and girls.   submitted by   /u/Thrumpwart [link]   [comments]

33
r/LocalLLaMA community 8d ago

Local AI World Model Part 2 - Deep NN to turn Images into Playable Characters, with prompt switching mid rollout

Last time when I posted on this subReddit to share my work, the response almost made me cry because a tiny demo got so many people talking about this In the past 2 months Ive been training a new model, but this time with actual text guidance. So a little about the older model.…

4
r/LocalLLaMA community 8d ago

GPT-6.1 Sol looped "leak" hints at nested models serving architecture

Hello llamas. I am posting this because I believe that, despite it being closed source models, the discussion will bring value to the local AI community. As many of you probably heard, GPT-6 Astra is speculated to be a looped transformer architecture that outputs a token after…

31
r/LocalLLaMA community 8d ago

EmbeddingGemma 2 is a best-in-class open model for natively multimodal embeddings

  submitted by   /u/sn2006gy [link]   [comments]

10
r/LocalLLaMA community 8d ago

Local AI ecosystem overview

Hey guys, it's Merve from Hugging Face! I've recently given a talk in a dev conference about llama.cpp + but also covering basic concepts like prefill vs decode, memory types, speculative decoding etc. you can use it if you feel like it and I appreciate if you can give…

33
r/LocalLLaMA community 8d ago

Aplomb 1: open-weights 5.3B decision model, 1M context, text/image/video/audio in one request, #1 among 4B models on the Decision Index

We released Aplomb 1 today, a 5.3B decision model with open weights. It reads up to 1M tokens of text, JSON, images, video and audio in a single request, and on our API it makes a decision on a 1M-token document in about 3 seconds, at $0.02 per 1M input tokens with free output…

36
r/LocalLLaMA community 8d ago

I trained a model to be wrong 98% of the time and 96% sure about it. It took three tries.

Meet Bev. She is a decision model (the Jev / Nimble kind: you give her a situation and a question, she gives a probability for each answer), fine-tuned on Qwen3.5-9B to pick the worst answer on purpose. Try her in your browser: https://huggingface.co/spaces/richardyoung/ask-bev…

11
r/LocalLLaMA community 8d ago

I gave a 21M model a 6.4B-parameter lookup table. It matches a 114M dense model and runs with the table on an SSD (RX 9070)

I spent the last few weeks on a hobby research project and just made it public. The idea isn't new (product-key memory, Lample et al. 2019, and Meta's "Memory Layers at Scale"): give a model a huge table of learned vectors and let it read only a few hundred of them per token. I…

19
r/LocalLLaMA community 8d ago

Introducing EmbeddingGemma 2: A best-in-class open model for natively multimodal embeddings | Google

https://huggingface.co/google/embeddinggemma-2 EmbeddingGemma 2 is an open multimodal embedding model built by Google DeepMind which maps text (incl. code), images, video, and audio inputs—and combinations thereof—into a single, unified 768-dimensional vector space. The model…

9
r/LocalLLaMA community 8d ago

EmbeddingGemma 2 running locally in-browser on WebGPU

Model: https://huggingface.co/google/embeddinggemma-2 Demo: https://huggingface.co/spaces/webml-community/embeddinggemma-2-webgpu   submitted by   /u/xenovatech [link]   [comments]

6
r/LocalLLaMA community 8d ago

54gb vram for 35$

Bought an old mining farm of a guy on avito (Russian eBay), guy had bought a garage a couple of years ago and it was sitting there for a while, found out it was a mining farm and put it up on there for sale for 5000 rub (~60 USD) since he wasn't sure if it works. I negotiated…

10
r/LocalLLaMA community 8d ago

google/embeddinggemma-2 · Hugging Face

EmbeddingGemma 2 is an open multimodal embedding model built by Google DeepMind which maps text (incl. code), images, video, and audio inputs—and combinations thereof—into a single, unified 768-dimensional vector space. The model has 740M total parameters, combining a 270M…

18
r/LocalLLaMA community 8d ago

Woman used claude as her diary - and got reported to the police for contents of her diary

  submitted by   /u/Timely_Impression_92 [link]   [comments]

4

Tested in Coding: Strata

Surface RTX Spark Dev Box: The Dev Box Built For Developers

[R] Split the Differences, Pool the Rest: Provably Efficient Multi-Objective Imitation

Qwen 3.8 Flash-Next on 64GB RAM + 16GB VRAM, worth it over a fully-loaded 3.6 35B-A3B?

llama.cpp on the stage

Kandinsky 6.0 video gen + video upscaler!

llama : add a GPU cache for MoE experts kept in host memory by am17an · Pull Request #29887 · ggml-org/llama.cpp

omni-d1 600M by Liquid AI running in the browser using WebGPU

d1-3B and d1-omni from LiquidAI

LiquidAI/d1-3B · Hugging Face

A week in Beijing and Shanghai with the people building AI in China

GPU Upgrade advice

Local Qwen 3.8 27B vs DeepSeek Flash API: Is local good enough?

feat: add GLM5Next MTP, optimize by pwilkin · Pull Request #29928 · ggml-org/llama.cpp

New LFM to be released today

Seeking upgrade advice

Micron Says NVHBM to Improve Profitability Even With Outsourced Base Die

cmpunlocker v0.5 just dropped, ECC support along with 4 extra SM unlocked for FREE, who needs a 64GB DGX Spark when you've got a CMP170hx right? 1.5TB/s memory BW vs 273 GB/s, all for less than 1/2 the price of a 64GB DGX Spark

Image-text retrieval with EmbeddingGemma 2's vision tower, running in the browser on WebGPU

A stock GLM-Edge-1.5B-Chat on a 4GB Galaxy A04e completed a real Amazon cart task

I re-trained the DFlash 2 drafter for Ternary Bonsai 2 27B: 2.2x on an L4 (3.2x on code edits with ngram lookup), 1.5x on a Mac, 1.2x in Chrome

Ramjet - mini altermative to nvidia dynamo

Single 3090 Qwen 27B user, considering buying 128GB of RAM because of the hype

New 35B-A3B model is comming on huggingface

Trained a ~20K LM (probably smallest) that can still write stories

Local RAG for a personal second brain: embedder and hybrid retrieval picks in 2026?

How far we’ve come

BenchLabs' Cagliostro V3.5 135M finally takes Hugging Face's SmolLM2-135M's first place on Open SLM Leaderboard with an Intelligence Index of 27.49

Why is ik_llama.cpp said to be faster than Mainline? On my hybrid multi-GPU rig, Mainline easily beats it

Ugh I didn't want to post this... Back to Qwen3.8 27B

Ruach Studio: a whole song studio around YuE2 on your own GPU. Score first, LoRA training, stems,remaster, DAW export.

Are Tesla K80s any good for inference?

MiMo V2.6 Pro almost matched GPT-6 Astra at figuring out what a user actually wants

I have about $4000, what's the best setup to get?

Qwen3.8-27B with just 12GB VRAM + 8GB RAM - ~18 tok/sec

pi-optchat: never compact again - endless chat as a memory tree

Europe rejoins the fight with Chonky! Mistral Large 4 Released, Open weights end of month, who’s ready?

How abliterated models can get you pwned

Local AI World Model Part 2 - Deep NN to turn Images into Playable Characters, with prompt switching mid rollout

GPT-6.1 Sol looped "leak" hints at nested models serving architecture

EmbeddingGemma 2 is a best-in-class open model for natively multimodal embeddings

Local AI ecosystem overview

Aplomb 1: open-weights 5.3B decision model, 1M context, text/image/video/audio in one request, #1 among 4B models on the Decision Index

I trained a model to be wrong 98% of the time and 96% sure about it. It took three tries.

I gave a 21M model a 6.4B-parameter lookup table. It matches a 114M dense model and runs with the table on an SSD (RX 9070)

Introducing EmbeddingGemma 2: A best-in-class open model for natively multimodal embeddings | Google

EmbeddingGemma 2 running locally in-browser on WebGPU

54gb vram for 35$

google/embeddinggemma-2 · Hugging Face

Woman used claude as her diary - and got reported to the police for contents of her diary