r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 8h ago

The AI race: China's open-weight models trail US closed models by just 7 months

Source: whatmodel.sh , an open source AI leaderboard I built (and launched here yesterday). Every point comes from published benchmark results: 22,113 of them across 987 benchmarks, from 1,011 sources, each one linked to where it was published. This chart uses data for the text…

35
r/LocalLLaMA community 9h ago

Anthropic Executive Says Open-Source AI Models Are Higher Risk

https://www.bloomberg.com/news/articles/2026-10-14/anthropic-executive-says-open-source-ai-models-are-higher-risk https://www.business-standard.com/technology/artificial-intelligence/open-source-ai-models-carry-more-cybersecurity-risks-anthropic-executive-126101400731_1.html…

25
r/LocalLLaMA community 9h ago

ComfyUI v0.40.0 released

  submitted by   /u/Fcking_Chuck [link]   [comments]

4
r/LocalLLaMA community 13h ago

So this is how they do it --> RTX 5090, once a consumer gaming card, is now on smugglers’ menu alongside cigarettes

  submitted by   /u/chemist_slime [link]   [comments]

7
r/LocalLLaMA community 14h ago

NVIDIA's Run:AI is now apparently under Apache-2.0 license.

  submitted by   /u/Ps3Dave [link]   [comments]

28
r/LocalLLaMA community 14h ago

Updated list of official MCP servers (300+), well-known and widely used projects only

We maintain an updated list of official MCP servers repo: 280+ well-known and widely used projects, all from the companies behind the products or actively maintained open source. Feedback and PRs welcome. github.com/VoltAgent/official-mcp-servers   submitted by  …

19
r/LocalLLaMA community 15h ago

Moving a running model's layers between GPUs mid-reply (8 ms, no re-prefill)

I've been working on an inference engine called jitllm. The idea came from my background in operating systems and JIT compilers: generate kernels for whatever hardware is there, and treat layers a bit like processes the engine can place and move around. In the video, a cloud VM…

35
r/LocalLLaMA community 17h ago

Discover AI: A Learning Guide - 51 hands-on labs for learning how AI actually works.

This is an offline app with 51 hands-on labs for learning how AI actually works, from neurons to agents and more... **A bit of insight** - 51 labs in 6 groups, from the basics (what a neuron is, gradient descent) through attention, RAG, agents, fine-tuning, quantization, serving…

32
r/LocalLLaMA community 20h ago

Fears Beyond Comprehension

Best advertisment for LocalAI lol   submitted by   /u/TheyCallMeDozer [link]   [comments]

14
r/LocalLLaMA community 20h ago

Real downsides of Strata?

Trying to read on what the real tested downsides have been but information seems to go both ways, some proclaim no loss, others pretty huge ones. I had some weird looping issue in their chat, and some engine port misunderstandings, seemingly the models likes to use ports already…

14
r/LocalLLaMA community 21h ago

My local Qwen3.8FN agent is significantly cheaper than cloud

tl;dr: running Qwen3.8FN Q4 locally is cheaper for me than every cloud model except Luna and the newer Haiku at mid-peak and peak electricity pricing where I live, and becomes the absolute cheapest model at off peak pricing. An agentic run that spans these different pricing…

21
r/LocalLLaMA community 21h ago

Claude CoWork for Local LLMs that runs on your Phone

Hi Friends, I am a solo developer. Ever since I had Qwen 3.8 running on my RTX4090, I had an itch to use it on my phone for everything. That gave me the idea to build SteerClaw Mobile - a complete agentic experience on the Phone where the entire agent loop runs on it. We just…

35 1
r/LocalLLaMA community 21h ago

Someone is running Qwen 27B at ~20 tok/s on a jailbroken PS5's GPU

  submitted by   /u/LatentSpacer [link]   [comments]

33
r/LocalLLaMA community 22h ago

Thank-you post for DeepSeek-V4-Flash-0731 with DSpark

DeepSeek guys, thank you so much! Don't give this up. mac setup [pi.dev] -> [ds 4 flash 0731 - brain] -> [qwen 3.6 35b a3b - body] - works perfect   submitted by   /u/AleksandrNikitin [link]   [comments]

20
r/LocalLLaMA community 23h ago

Character.AI files for bankruptcy, to sell some assets to Disney for $20 million

Are we seeing early signs of the bubble popping?   submitted by   /u/jld1532 [link]   [comments]

32
r/LocalLLaMA community 1d ago

Why AnythingLLM is moving from Ollama to Lemonade and llama.cpp

I'm excited to see our Lemonade embedded SDK getting used out there!   submitted by   /u/jfowers_amd [link]   [comments]

7
r/LocalLLaMA community 1d ago

Instead of waiting for an M5 Ultra 512 GB: pairing a new M5 Ultra 256 GB with my M3 Ultra 512 GB for disaggregated prefill/decode (GLM-5.3-Flash, MLX, Thunderbolt 5 RDMA)

The problem I'm solving: I have an M3 Ultra 512 GB and the memory is still the reason to own it, but prefill on long prompts is slow. A 17.5k-token prompt takes 41 s before the first token. The M5 Ultra fixes prefill but ships with 256 GB max right now, and an M5 Ultra 512 GB…

18
r/LocalLLaMA community 1d ago

"New AI fake text generator may be too dangerous to release, say creators" (2019 Guardian headline)

I'm sure some here read this at the time and remember, but I recently found this hilarious Guardian article from 2019 through Wikipedia: https://www.theguardian.com/technology/2019/feb/14/elon-musk-backed-ai-writes-convincing-news-fiction Some choice quotes: "OpenAI ... says its…

27
r/LocalLLaMA community 1d ago

LiquidAI/LFM2.5-24B-A2B

Gguf's: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B-GGUF Free Demo: https://playground.liquid.ai "LFM2.5-24B-A2B is a 24B-parameter reasoning model with only 2B active parameters, built for fast, private agents on-device . It is the largest and final model in the LFM2.5…

24
r/LocalLLaMA community 1d ago

Tencent open-sourced Agent Memory and it runs 100% locally on SQLite

  submitted by   /u/AdamDhahabi [link]   [comments]

35
r/LocalLLaMA community 1d ago

PSA for 3090 FE owners: Running LLMs overheats your VRAM

https://preview.redd.it/p6i9qifyy8vh1.png?width=2069&format=png&auto=webp&s=2b1e3c7c4fc95f12f0f7d417a4d21410aebfbd35 The 3090 Founders edition has a reputation for hot VRAM. Since LLM token generation maxes out VRAM's read capacity, I checked VRAM temps with Nvidia's stock…

37
r/LocalLLaMA community 1d ago

4x 6000 Pro for the company

Here is how i serve 40 people (10 of them developers) with local AI services. n8n for automatisation whisper and Vexa AI, a bot joins Teams and makes transcripts, n8n later extracts decisions and todos OpenWebUI, ChatGPT-like webinterface for everyone, most agentic features are…

31
r/LocalLLaMA community 1d ago

Netflix’s documentary of Huggingface discussion

It just popped on my feed. I watched the teaser in the Netflix app and a few things jumped out The constant use of anthromorphism using terms like altruism and sacrifice when in engineering literally there are terms like kill, killall, pkill for terminating processes, therefore…

15
r/LocalLLaMA community 1d ago

Basalt: Fast Blackwell inference engine for Qwen 3.8 Flash Next (and how we got vision for free)

Dynamic vision, in action Basalt is an inference engine for 3.8 Flash Next, forked from Strata and heavily tuned for Blackwell. It supports vision, and for this to work you need to have the image encoder either in VRAM or CPU all the time, having it on VRAM takes valuable space…

6
r/LocalLLaMA community 1d ago

An 80B MoE (Qwen3-Next, 48 GB GGUF) answering in a Chrome tab: llama.cpp's WebGPU backend, experts paged in from disk

Follow-up to last week's post about streaming MoE experts from disk in a browser tab. That ran on hand-written engines for two models. This week the same disk tier sits under llama.cpp's own WebGPU backend, so it works for the MoE models llama.cpp already runs, with no per-model…

16
r/LocalLLaMA community 1d ago

Built an open-source Cotypist/Granola hybrid: autocomplete on Gemma 4 E2B (~50 ms/suggestion on M4 Max) using your own meeting notes as context

I built a desktop app harness that keeps local meeting notes and can use them to autocomplete my typing as well as complement my dictation with additional context via reading my screen or using the previous meetings notes. It's been a bit of a struggle to get it to work well and…

18
r/LocalLLaMA community 1d ago

Is everyone in here buying up smart TVs now?

Anthropic says we can run Gemma4 26b a4b with only 14 smart tvs.   submitted by   /u/SporksInjected [link]   [comments]

37
r/LocalLLaMA community 1d ago

Horsey challenge | Strata Fork | Qwen 3.8 FN NVFP4

So I wanted to try the horsey challenge myself on the latest engine i been testing which is a strata fork by sergqwer. https://github.com/sergqwer/strata-nvfp4 . I seen a post earlier and seen i believe it was a iq3 i think and it was pretty dope so i thought i try out the…

34
r/LocalLLaMA community 1d ago

Update: Converting dense models into Mixture-of-Experts

This is an update to my previous post which can be found here: https://www.reddit.com/r/LocalLLaMA/s/n7sgRZZc4z Since my last post I have continued working on the conversion process and also converting new models. My latest model is the Qwen2.5-1.5B-MoE-A0.9B . I have also…

37
r/LocalLLaMA community 1d ago

It's been too quiet on the other side of the Pacific

  submitted by   /u/Nunki08 [link]   [comments]

28
r/LocalLLaMA community 1d ago

Got 2x DGX Sparks. Excited and confused!

Got this couple. It was difficult purchase in this economy but paid only roughly $6100 equivalent for each in my country. I justified the purchase for learning (so I try to convince myself) and coding workflow improvements. I started looking into recipes for GLM 5.3 flash but…

11
r/LocalLLaMA community 1d ago

Based on advice from LocalLLama implemented a n+2 lookahead prediction for ssd streaming Qwen Flash Next - and it worked!

My project is optimising qwen flash next on Mac Mini m5 64gb through SSD streaming. I recently asked LocalLLama for guidance on how to optimise token output. Today a popular method to do this is a lookahead guesser which fetches experts one layer ahead of the current layer. One…

13
r/LocalLLaMA community 1d ago

SonicBoom: 52 tok/s decode on a laptop RTX 3050 (llama.cpp: 45) with the same quantized kernels, the difference is a graph compiler. No Python.

I’ve been building SonicBoom, a native tensor runtime written in C++/CUDA with a C ABI and no Python anywhere in the stack. I just tagged v0.0.1, and the first target is local LLM inference on consumer GPUs. Numbers (Gemma 4 E4B-it Q3_K_M on a laptop RTX 3050 8 GB, both runtimes…

12
r/LocalLLaMA community 1d ago

Sipeed announces MaixStation, comparable to Strix Halo and DGX Spark

  submitted by   /u/Marcuss2 [link]   [comments]

11
r/LocalLLaMA community 1d ago

Built a VRAM scheduler (vramble) to automate model swapping on my local AI workstation

I set up a dedicated workstation for local AI experimentation with various models and tools. But every single one of them wants all the available VRAM. To fix this, I built a request dispatching layer called vramble . It automatically loads and unloads models and manages the…

32
r/LocalLLaMA community 1d ago

Anyone feel like reverse-engineering CUDA?

I've been reading about all these modders using AI to reverse-engineer games and software. I've been trying out photocraft and gridcraft and have just watched a GTA5 mod featuring Sonic. It's pretty damn impressive. As a struggling AMD Radeon owner I'd love to see AI generated…

24
r/LocalLLaMA community 1d ago

Hope the software support is good, but -> Intel Crescent Island Squeezes 1.3 PFLOPs Of FP8 Into Just 350W As D270 Targets The Competition With Value-Oriented LPDDR5X Up To 480 GB

  submitted by   /u/chemist_slime [link]   [comments]

19
r/LocalLLaMA community 1d ago

NVIDIA’s 9-Year-Old Tesla V100 Still Packs Some Inference Muscle; 4 GPUs On An NVLink Board Can Outperform M5 Ultra, DGX Spark In Various AI Benchmarks By Up To 230%

Didn't realize the V100 were still so fast, anyone have first hand experience on these?   submitted by   /u/chemist_slime [link]   [comments]

14
r/LocalLLaMA community 1d ago

I clustered two old Android phones to run a 27B LLM locally. Bypassed RAM limits via C++ RPC split and Hadamard Ternary math (No overheating)

I saw a lot of posts here struggling to run 27B models on 8GB VRAM or complaining about phones turning into space heaters. I hit the exact same OOM wall on my phone, so I decided to build a hardware-split workaround. I wrote a custom C++ engine (fork of llama.rn ) to cluster two…

9
r/LocalLLaMA community 1d ago

How far are we from a local model being able to optimize it's own runtime stack?

A lot of projects have come up recently consisting of specific hardware config specialized forks of existing conferencing engines, but they all seem to be made using Claude or OpenAI's biggest frontier models. So given that open source is about 4-6 months behind, can we…

32
r/LocalLLaMA community 1d ago

Anthropic is doing free publicity for GLM again

https://www.anthropic.com/research/distillation-defenses   submitted by   /u/No-Point1424 [link]   [comments]

29
r/LocalLLaMA community 1d ago

Qwen 3.8 Flash Next Q3 S - Benchmark creation

After running a benchmark test on Exo in Opencode and being amazed by the result, I thought I'd try other models on Opencode. Many were surprisingly terrible. Especially some mid level models. Earlier posts:…

14
r/LocalLLaMA community 1d ago

Intel "Crescent Island" GPU Gets D-Series Name, Supports Up to 480 GB of LPDDR5X

The newly named D-Series GPU has up to 480GB of LPDDR5X memory, 1.5TB/s of memory bandwidth Intel rates it for up to 1.3 PetaFLOPS of FP8 performance, with support for formats ranging from FP4 to FP64. It features a total of 32 Xe3P cores, 256 XMX engines, 32 MB of unified L2…

16
r/LocalLLaMA community 1d ago

Explore DNA with DeepMind’s AlphaGenome and in-browser MedGemma 1.5 4B via WebGPU

Hey everyone, Sharing an educational tool called AskCyph Health Explorer to explore how single-letter DNA mutations lead to disease, how they alter 3D anatomy, inheritance probabilities, and available treatments. To make it interactive while protecting user privacy, the…

33
r/LocalLLaMA community 1d ago

llama.cpp optimizations for specific hardware and models

I've made a few posts recently, just talking about the hardware I've got and the model I use. Pair of P100s, Unsloth Qwen3.6 35b A3B UD Q4 K XL . There's been a ton of talk of model and hardware specific inference engines showing crazy performance numbers on models and hardware…

27
r/LocalLLaMA community 2d ago

Running unsloth/Qwen3.8-27B-UD-IQ4_XS 262k q4_0 KV + vision on 16GB VRAM @ 80t/s and 1500pp

I get a lot of hate whenever I tell people I run a Q4 quant of Qwen3.8 27B with 262k KV 80t/s with vision, all while running 2 screens on windows on the same card. People seem to think I am lying or making it up. I am really just trying to help people. I am saddened whenever I…

31
r/LocalLLaMA community 2d ago

Once again anthropic doing free marketing for open weight models

  submitted by   /u/kubblegum [link]   [comments]

6
r/LocalLLaMA community 2d ago

It's been 2 months

  submitted by   /u/Comfortable-Rock-498 [link]   [comments]

10
r/LocalLLaMA community 2d ago

Qwen3.8-27B vs Frontier vs Others

Here're results from my final test of coding, coding (panda/numpy), data analysis, data-driven decision-making, and local RAG domains I interact with most often. This test, which is refined from a 500+ pack, now provides a differentiating signal between models after having…

24
r/LocalLLaMA community 2d ago

openbmb/MiniCPM-V-4.7-1B

Gguf's: https://huggingface.co/openbmb/MiniCPM-V-4.7-1B-gguf " MiniCPM-V 4.7 is the latest model in the MiniCPM-V series, available in a 1B variant. It builds on the family's single-image, multi-image, and video understanding capabilities, combining stronger foundation…

32

The AI race: China's open-weight models trail US closed models by just 7 months

Anthropic Executive Says Open-Source AI Models Are Higher Risk

ComfyUI v0.40.0 released

So this is how they do it --> RTX 5090, once a consumer gaming card, is now on smugglers’ menu alongside cigarettes

NVIDIA's Run:AI is now apparently under Apache-2.0 license.

Updated list of official MCP servers (300+), well-known and widely used projects only

Moving a running model's layers between GPUs mid-reply (8 ms, no re-prefill)

Discover AI: A Learning Guide - 51 hands-on labs for learning how AI actually works.

Fears Beyond Comprehension

Real downsides of Strata?

My local Qwen3.8FN agent is significantly cheaper than cloud

Claude CoWork for Local LLMs that runs on your Phone

Someone is running Qwen 27B at ~20 tok/s on a jailbroken PS5's GPU

Thank-you post for DeepSeek-V4-Flash-0731 with DSpark

Character.AI files for bankruptcy, to sell some assets to Disney for $20 million

Why AnythingLLM is moving from Ollama to Lemonade and llama.cpp

Instead of waiting for an M5 Ultra 512 GB: pairing a new M5 Ultra 256 GB with my M3 Ultra 512 GB for disaggregated prefill/decode (GLM-5.3-Flash, MLX, Thunderbolt 5 RDMA)

"New AI fake text generator may be too dangerous to release, say creators" (2019 Guardian headline)

LiquidAI/LFM2.5-24B-A2B

Tencent open-sourced Agent Memory and it runs 100% locally on SQLite

PSA for 3090 FE owners: Running LLMs overheats your VRAM

4x 6000 Pro for the company

Netflix’s documentary of Huggingface discussion

Basalt: Fast Blackwell inference engine for Qwen 3.8 Flash Next (and how we got vision for free)

An 80B MoE (Qwen3-Next, 48 GB GGUF) answering in a Chrome tab: llama.cpp's WebGPU backend, experts paged in from disk

Built an open-source Cotypist/Granola hybrid: autocomplete on Gemma 4 E2B (~50 ms/suggestion on M4 Max) using your own meeting notes as context

Is everyone in here buying up smart TVs now?

Horsey challenge | Strata Fork | Qwen 3.8 FN NVFP4

Update: Converting dense models into Mixture-of-Experts

It's been too quiet on the other side of the Pacific

Got 2x DGX Sparks. Excited and confused!

Based on advice from LocalLLama implemented a n+2 lookahead prediction for ssd streaming Qwen Flash Next - and it worked!

SonicBoom: 52 tok/s decode on a laptop RTX 3050 (llama.cpp: 45) with the same quantized kernels, the difference is a graph compiler. No Python.

Sipeed announces MaixStation, comparable to Strix Halo and DGX Spark

Built a VRAM scheduler (vramble) to automate model swapping on my local AI workstation

Anyone feel like reverse-engineering CUDA?

Hope the software support is good, but -> Intel Crescent Island Squeezes 1.3 PFLOPs Of FP8 Into Just 350W As D270 Targets The Competition With Value-Oriented LPDDR5X Up To 480 GB

NVIDIA’s 9-Year-Old Tesla V100 Still Packs Some Inference Muscle; 4 GPUs On An NVLink Board Can Outperform M5 Ultra, DGX Spark In Various AI Benchmarks By Up To 230%

I clustered two old Android phones to run a 27B LLM locally. Bypassed RAM limits via C++ RPC split and Hadamard Ternary math (No overheating)

How far are we from a local model being able to optimize it's own runtime stack?

Anthropic is doing free publicity for GLM again

Qwen 3.8 Flash Next Q3 S - Benchmark creation

Intel "Crescent Island" GPU Gets D-Series Name, Supports Up to 480 GB of LPDDR5X

Explore DNA with DeepMind’s AlphaGenome and in-browser MedGemma 1.5 4B via WebGPU

llama.cpp optimizations for specific hardware and models

Running unsloth/Qwen3.8-27B-UD-IQ4_XS 262k q4_0 KV + vision on 16GB VRAM @ 80t/s and 1500pp

Once again anthropic doing free marketing for open weight models

It's been 2 months

Qwen3.8-27B vs Frontier vs Others

openbmb/MiniCPM-V-4.7-1B