r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 8d ago

[Paper] FactorEngram: Factorized N-gram Memory with Basis-Level Gating for Language Models

Lookup-based memory has been a promising way to scale the parameters of large language models (LLMs). It retrieves learned representations of local token patterns, such as n-grams, instead of reconstructing them through successive layers of computation. However, existing designs…

5
r/LocalLLaMA community 8d ago

Qwen3.8-Flash-Next on Strata

Hey! 👋 I have released an official support for Strix Halo machines on Strata for Qwen3.8-Flash-Next. Currently numbers are the best on long context decode and ppts using typical Unsloth’s Q4 and GSQ-RCO model weights. Can go up to 1M context length without big speed loss.…

32
r/LocalLLaMA community 8d ago

Mistral-Large-4.0 weights excepted in ~25 days

  submitted by   /u/paf1138 [link]   [comments]

4
r/LocalLLaMA community 8d ago

Q (@qtnx_) on X - Mistral Large 4 is still doing RL runs, keep seeing improvements (vs preview version). Release at the end of the month

  submitted by   /u/DerpSenpai [link]   [comments]

18
r/LocalLLaMA community 8d ago

Mistral releasing new open weights model "Le Chonk"

LeChonk-1T-A49B Weights to be released by the end of the month. This probably means the fat kitten is dead. Also: gotta love the release movie.   submitted by   /u/wapswaps [link]   [comments]

32
r/LocalLLaMA community 8d ago

Introducing Mistral Large 4 (Preview)

  submitted by   /u/Few_Painter_5588 [link]   [comments]

38
r/LocalLLaMA community 8d ago

Le Chonk strikes back.

  submitted by   /u/atape_1 [link]   [comments]

17
r/LocalLLaMA community 8d ago

[Paper] WaveFront Decoding: Parallelized Self-Speculative Decoding for Looped Language Models

Looped language models repeatedly apply a weight-shared block to increase effective depth without increasing parameter count, but the resulting T sequential recurrent-block calls per generated token substantially increase decoding latency. To address the issue, we introduce…

11
r/LocalLLaMA community 8d ago

Mistral Large 4 benchmarks

Just dropped.   submitted by   /u/Informal-Trouble2183 [link]   [comments]

24
r/LocalLLaMA community 8d ago

Meet Mistral Large 4, aka Le Chonk.

  submitted by   /u/cpldcpu [link]   [comments]

20
r/LocalLLaMA community 8d ago

Qwen 4 apparently coming out at the end of October

Hey All, I spoke to a 0-day partner of Alibaba today and he casually mentioned (didnt know if he was allowed to) that Qwen 4 is apparently planned for the end of October. To me, this is way faster than expected as there was quite a gap between 3.6 and 3.8. I tried to get more…

22
r/LocalLLaMA community 8d ago

What happens when a LLM watches its own context window run out?

I made Terminal Soliloquy , a terminal artwork that connects to llama.cpp and displays a model's monologue as its context window fills. It has a retro phosphor look, and runs in a terminal window. I'm actually running it full-screen on a Raspberry Pi display inside an old 1960s…

6
r/LocalLLaMA community 8d ago

Microsoft confirms OpenAI has been using Looped Transformers in the GPT-6 series

Microsoft confirms on publicly accessible web page that OpenAI has been using Looped Transformers in the GPT-6 series, proving The Information's reporting was correct all along. GPT-6.1 Sol uses 2 inference passes, with a passing mention of "instead of three". For those confused…

35
r/LocalLLaMA community 8d ago

Story time: Qwen3.8-Flash-Next on my Strix Halo laptop vs Claude Opus 5.5 on the same feature

For the last few weeks most of my coding has been done locally with Qwen3.8-Flash-Next, so I gave it and Opus 5.5 the same high complexity feature to build on LlamaStash (a complex and large Rust project) and compared the results. Setup : ASUS ROG Flow Z13 (Strix Halo, 128GB),…

21
r/LocalLLaMA community 8d ago

Tencent releases Octop, a self-hosted AI assistant

Octop is an open-source, self-hosted AI assistant. Through its multi-agent architecture, it builds an intelligent environment that is both independent and collaborative for teams, families, and individuals. Best of all, it runs entirely on your machine, the fully self-hosted…

5
r/LocalLLaMA community 8d ago

NInfer6000 - Qwen 3.8 Flash Next @ 400 tg/s & 13K pp/s

I've tinkered some more with my fork of NInfer for the Qwen 3.8 Flash Next model on RTX6000, and I've just hit 400tg/s under ideal conditions with it, so I thought it was worth a share. This is NVFP4 MoE's and the rest is either 16-bit or 8-bit, so it uses full 96GB VRAM and…

24
r/LocalLLaMA community 8d ago

Mistral CEO says new AI model beats Chinese ones in some areas

  submitted by   /u/tengo_harambe [link]   [comments]

17
r/LocalLLaMA community 8d ago

ComfyUI v0.39.0 released

  submitted by   /u/Fcking_Chuck [link]   [comments]

25
r/LocalLLaMA community 8d ago

I made a Chrome extension that filters your YouTube feed with a small local model running in the browser (WebGPU, no server)

My YouTube feed was mostly songs, pranks and celebrity clips, so I built a filter that judges each video title with a small decision model running inside Chrome. Nothing is sent anywhere: no server, no API key, no account, and after one download it works offline. What it does -…

37
r/LocalLLaMA community 8d ago

Why 38% of AI Agent container escapes didn't need kernel 0-days: Analysis of 109 empirical incidents (Open Dataset + Defense Harness)

Over the past several months, we conducted an empirical post-mortem investigation into 109 autonomous AI agent security incidents (cataloged with 193 falsification criteria across tool-use and multi-agent systems). One of the most striking patterns in the dataset: In 38% of…

30
r/LocalLLaMA community 8d ago

We’re using GLM-5.3 Flash instead of frontier models on a massive production codebase

At my company, we’re using GLM-5.3 Flash internally for software engineering work, and I’ve been genuinely impressed by it. I work in a very large production environment with projects totaling **millions of lines of code**, and we’re not relying on frontier models for this…

23
r/LocalLLaMA community 8d ago

unsloth/Qwen3.8-Flash-Next-GGUF is being updated

Looks like Unsloth is updating https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF to work with llama.cpp, so hopefully this will resolve the issue of having two different GGUF versions of Qwen Flash Next.   submitted by   /u/jacek2023 [link]   [comments]

26
r/LocalLLaMA community 8d ago

NVIDIA board member Mark Stevens sold $946.63M in NVIDIA (NVDA) shares in September

NVIDIA board member Mark Stevens has executed a massive series of stock sales. SEC regulatory filings reveal that Stevens sold approximately $946.63 million worth of Nvidia (NVDA) common stock over three distinct waves. He sold ~13.3% of his total NVDA position.…

28
r/LocalLLaMA community 8d ago

When will we get another good American open weights model?

Every time I check the top models on HF, I basically feel like this:   submitted by   /u/cyber_burr [link]   [comments]

8
r/LocalLLaMA community 8d ago

Set your P(doom) on HF

  submitted by   /u/reto-wyss [link]   [comments]

38
r/LocalLLaMA community 8d ago

Qwen 3.8 27b just feels… ok?

I’ve seen posts here raving about how good Qwen 3.8 27b is. The benchmarks look incredible, and all the online discourse seems to deem it the best local model. I have 32GB VRAM and run Unsloth’s Q6 version with OpenCode. For small tasks, it feels fine. I range 30-40 t/s decode…

15
r/LocalLLaMA community 9d ago

k_llama.cpp MoE Optimizations: Expert Residency, Hybrid CPU/GPU Execution, Q2_0 Support

Finally finished my fork: https://github.com/IceFog72/ik_llama.cpp Nothing else I wanted to add/try currently works In short, it now has: Bandwidth-adaptive CPU/GPU execution Shared expert-residency ideas from https://arxiv.org/pdf/2608.16157 integrated into this fork's…

21
r/LocalLLaMA community 9d ago

A benchmark for LLMs playing Civilization V. GLM-5.3 is ahead of Opus-5.5, and Qwen-3.8-27B holds up surprisingly well.

A while ago, I posted here getting OSS-120B and GLM-4.6 playing full games of Civilization V . Since then, models have moved pretty far, and we wanted a better understanding about models' capabilities playing the game. Introducing the controlled version of CivBench on newer…

14
r/LocalLLaMA community 9d ago

PewDiePie getting banned twice by OpenAI while making a local model is top-tier comedy 💀

So PewDiePie decides to fine-tune a local AI model called Ajax on his own computer. Pretty normal stuff for local model fans. To make his dataset, he uses OpenAI's API. OpenAI catches him using their outputs to train another model, flags his account for breaking their terms, and…

30
r/LocalLLaMA community 9d ago

When Redditors come in here and ask why we run LLMs, this is why: Big AI is watching.

Anthropic Reports Florida Woman's Claude 'Diary' Threat to Law Enforcement And this time it wasn't the AI model that made the LEO referral. It was the "human review team". The frontier AI companies are watching your input. And people say "Well I'm not interesting or important…

15
r/LocalLLaMA community 9d ago

nokia-applied-research/AnyJev: Turn any LLM into a Jev-style decision model

Haven't seen posted here. The research project matters because it proves you can pull trustworthy decisions straight out of an LLM's internal state in a single forward pass (no generation loop, no parsing, no fine-tuning) essentially turning a slow chatbot into a fast and cheap,…

19
r/LocalLLaMA community 9d ago

We swapped AdamW's optimizer states for a Fast Fourier Transform (FFT) to cut VRAM in half. Anyone else trying non-quantization methods?

Hey everyone, Like most of you, we have been fighting constant OOM errors while trying to fine-tune 8B and 70B models on consumer GPUs. The AdamW optimizer states are always the biggest bottleneck. We didn't want to rely on aggressive 8-bit quantization because we were seeing…

15
r/LocalLLaMA community 9d ago

llama.cpp v0.6.0 released with MTP speculative decoding for Qwen4Exp and lots more

  submitted by   /u/vexatious-big [link]   [comments]

12
r/LocalLLaMA community 9d ago

Y'all this is a sexy paper; context language models

Paper linky - Context Language Models The central idea of the paper is incredibly simple. Give a model the ability to edit its context on-the-go like a file has major benefits on task performance, context management (memory) and even computational efficiency (both wall clock and…

13
r/LocalLLaMA community 9d ago

Whistle: speech to text in a 16.9MB file

Hey all, we designed Cactus Whistle, an ASR model for ultra-small devices. It's not perfect, but mostly beats Whisper base with 9x less file size and 6x speed. Whistle supports English, German, French, Spanish, Italian, Dutch and Polish. Remember, the goal at Cactus Compute…

20
r/LocalLLaMA community 9d ago

How is it possible that qwen 27b is so good? When GPT 4o had a trillion parameters and was worse?

Picture from a post in r/amodei . People were praising qwen and I'm just wondering, what kind of new technologies are at play here? Does qwen just have "better" pre training data? That's more high quality?   submitted by   /u/SignificantZebra5883 [link]   [comments]

13
r/LocalLLaMA community 9d ago

Make no mistake, selling 64 GB DGX Spark variants at the same cost as the original 128 GB is straight drug dealer behavior.

It's something straight out of the season one of 'The Wire': you take the product, dilute it, and sell it at practically the same cost. It's some "Stringer" Bell shit. We should call the 64gbs "Stepped-ons" from now on.   submitted by   /u/blacklandothegambler [link]…

35
r/LocalLLaMA community 9d ago

SkyIsNotGreen/Scion-35B-A3B · Hugging Face - Ternary MoE

GitHub (llama.cpp fork) : https://github.com/sky-is-green/prism-ml-llama.cpp (Prebuilt binaries there to try instantly) Draft model : https://huggingface.co/SkyIsNotGreen/Scion-35B-A3B-mtp-drafter   submitted by   /u/pmttyji [link]   [comments]

8
r/LocalLLaMA community 9d ago

Speakrail - a low-latency fully-local voice assistant that runs on a single RTX 4090

tl;dr: I created a fully-local open-source full-duplex voice agent that rivals GPT-Live on some benchmarks. It uses Voxtral Realtime with a turn-taking head, a microturn-finetuned Gemma 4 12B and Breeze TTS 2 under the hood. Go try it out: https://github.com/speakrail/speakrail…

21
r/LocalLLaMA community 9d ago

Finetuned 1.5B Qwen to generate bash commands at gpt-4o level using 400k synthetic examples + Fully opensource finetune dataset

Purely a hobby side project to see how far I can push a really small model, using (mostly) automated training pipelines Full synthetic data: https://huggingface.co/datasets/dirac-run/ec-training-data Models: https://huggingface.co/dirac-run/ec-1.5b-gguf and…

15
r/LocalLLaMA community 9d ago

Smallest Jev-like model

TinyDecide is 10M Jev-like mode with 10M parameters and fits in just ~6MB. Smaller than every model on the Decision Index leaderboard and it punches way above its size . It runs almost anywhere: in the browser, Node.js, Python, Rust, and even on an ESP32.…

7
r/LocalLLaMA community 9d ago

Qwen3.8-Flash-Next (125B) on a single Strix Halo mini PC: 44-59 tok/s with speculative decoding, ~1,400 tok/s prefill, engine is open

Hey all. We've spent the last weeks getting Qwen3.8-Flash-Next (125B MoE, 6B active) to run properly on one AMD Strix Halo box (Ryzen AI Max+ 395, 128 GB). Tonight we're releasing both the 95 GB EXL3 weights and a new version of Kyojin, our inference engine (built on ExLlamaV3,…

9
r/LocalLLaMA community 9d ago

Qwen3.8-27b appreciation moment

q3.8-27b q3_k_xl this thing have done everything i possibly needed from him he wired up my openwebui spawned trillium service fix all my bugs set up pi-web-ui and debug my cloudflare tunnel it even spawned smaller LLMs to make the LLM use the toole he created to ensure it will…

20
r/LocalLLaMA community 9d ago

I built an open-source real-time Japanese anime subtitle & translation engine powered by Whisper-Large-v3 + Groq / DeepSeek

Hey r/LocalLLaMA , Like many anime fans, I've always been frustrated by traditional MT engines (like Google Translate or base DeepL) when dealing with raw Japanese anime: - They completely butcher Japanese honorifics, sentence-ending particles ( -tteba , -zo , -desu wa ), and…

25
r/LocalLLaMA community 9d ago

ItoTTS: two natural English voices in 4.89 MB for a $5 ESP32-S3

Hi everyone! I'm part of the Lokutor team. Last week we presented Oído here, and the response was amazing. We've received dozens of videos and messages from you guys saying you love it. Thank you! Now we're back with the next part of our plan: ItoTTS, a natural-sounding,…

26
r/LocalLLaMA community 9d ago

M5 Ultra 256 running GLM 5.3 Flash 68.8 tok/s

Running oQ4e+MTP on oMLX 0.7.0, with still more to optimize. Prefill is 1,878 toks. I saw some other benchmarks below what id expect so i figured I would share.   submitted by   /u/cryotic [link]   [comments]

37
r/LocalLLaMA community 9d ago

Agens Volundr 32B Preview: our small team's first model on our own hybrid architecture. Only 18 of 72 layers keep a KV cache (Apache-2.0)

Hi r/LocalLLaMA . I'm on the team at Blockway, a small team in Hong Kong (disclosure: this is our model). Today we released Agens Volundr 32B Preview, the first model built on our own hybrid architecture. We trained it on limited compute, it isn't perfect, and we'd rather tell…

5
r/LocalLLaMA community 9d ago

MoE SSD streaming on a 64 GB Mac mini: GPU still waits 27% of decode on experts. Ideas?

Looking for some assistance /ideation. I am running qwen flash next q4 in my Mac mini m5 64gb. QFN doesn’t fit so this is done by having as many experts hot in cache as possible and streaming in the rest from ssd. I’m getting 17.5 tks decode and 390 tks pp. Have done a bunch of…

32
r/LocalLLaMA community 9d ago

Which model, which harness? I have data for you.

I keep testing harness & model setups on real tasks: basic math, vision, computer use (reading emails, navigating stores) and coding. I created a benchmark and a leaderboard. You can see it here: https://airbench.ai/leaderboard?k=poL It measure capabilities (a % of sucess on the…

9
r/LocalLLaMA community 9d ago

Clef Flash plays Snake in Real Time on RTX 5080

The cool part is No training was needed. No hacking of the game state or algorithms needed Just simple instructions about what the snake can see, etc, and it can play in real time. 135ms is the turn limit of Google snake, so basically could be a human playing. Ofc, it could be…

10

[Paper] FactorEngram: Factorized N-gram Memory with Basis-Level Gating for Language Models

Qwen3.8-Flash-Next on Strata

Mistral-Large-4.0 weights excepted in ~25 days

Q (@qtnx_) on X - Mistral Large 4 is still doing RL runs, keep seeing improvements (vs preview version). Release at the end of the month

Mistral releasing new open weights model "Le Chonk"

Introducing Mistral Large 4 (Preview)

Le Chonk strikes back.

[Paper] WaveFront Decoding: Parallelized Self-Speculative Decoding for Looped Language Models

Mistral Large 4 benchmarks

Meet Mistral Large 4, aka Le Chonk.

Qwen 4 apparently coming out at the end of October

What happens when a LLM watches its own context window run out?

Microsoft confirms OpenAI has been using Looped Transformers in the GPT-6 series

Story time: Qwen3.8-Flash-Next on my Strix Halo laptop vs Claude Opus 5.5 on the same feature

Tencent releases Octop, a self-hosted AI assistant

NInfer6000 - Qwen 3.8 Flash Next @ 400 tg/s & 13K pp/s

Mistral CEO says new AI model beats Chinese ones in some areas

ComfyUI v0.39.0 released

I made a Chrome extension that filters your YouTube feed with a small local model running in the browser (WebGPU, no server)

Why 38% of AI Agent container escapes didn't need kernel 0-days: Analysis of 109 empirical incidents (Open Dataset + Defense Harness)

We’re using GLM-5.3 Flash instead of frontier models on a massive production codebase

unsloth/Qwen3.8-Flash-Next-GGUF is being updated

NVIDIA board member Mark Stevens sold $946.63M in NVIDIA (NVDA) shares in September

When will we get another good American open weights model?

Set your P(doom) on HF

Qwen 3.8 27b just feels… ok?

k_llama.cpp MoE Optimizations: Expert Residency, Hybrid CPU/GPU Execution, Q2_0 Support

A benchmark for LLMs playing Civilization V. GLM-5.3 is ahead of Opus-5.5, and Qwen-3.8-27B holds up surprisingly well.

PewDiePie getting banned twice by OpenAI while making a local model is top-tier comedy 💀

When Redditors come in here and ask why we run LLMs, this is why: Big AI is watching.

nokia-applied-research/AnyJev: Turn any LLM into a Jev-style decision model

We swapped AdamW's optimizer states for a Fast Fourier Transform (FFT) to cut VRAM in half. Anyone else trying non-quantization methods?

llama.cpp v0.6.0 released with MTP speculative decoding for Qwen4Exp and lots more

Y'all this is a sexy paper; context language models

Whistle: speech to text in a 16.9MB file

How is it possible that qwen 27b is so good? When GPT 4o had a trillion parameters and was worse?

Make no mistake, selling 64 GB DGX Spark variants at the same cost as the original 128 GB is straight drug dealer behavior.

SkyIsNotGreen/Scion-35B-A3B · Hugging Face - Ternary MoE

Speakrail - a low-latency fully-local voice assistant that runs on a single RTX 4090

Finetuned 1.5B Qwen to generate bash commands at gpt-4o level using 400k synthetic examples + Fully opensource finetune dataset

Smallest Jev-like model

Qwen3.8-Flash-Next (125B) on a single Strix Halo mini PC: 44-59 tok/s with speculative decoding, ~1,400 tok/s prefill, engine is open

Qwen3.8-27b appreciation moment

I built an open-source real-time Japanese anime subtitle & translation engine powered by Whisper-Large-v3 + Groq / DeepSeek

ItoTTS: two natural English voices in 4.89 MB for a $5 ESP32-S3

M5 Ultra 256 running GLM 5.3 Flash 68.8 tok/s

Agens Volundr 32B Preview: our small team's first model on our own hybrid architecture. Only 18 of 72 layers keep a KV cache (Apache-2.0)

MoE SSD streaming on a 64 GB Mac mini: GPU still waits 27% of decode on experts. Ideas?

Which model, which harness? I have data for you.

Clef Flash plays Snake in Real Time on RTX 5080