r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 12d ago

Bought 2x r9700, 5090 is now 7k and 6000 pro is at 13.5k, best option for 64 gb vram under 4k

after being frustrated with nvidia proces, I went with asrock r9700, not even dgx spark even they are at 7k now, did I make a mistake?   submitted by   /u/AppropriatePush6262 [link]   [comments]

20
r/LocalLLaMA community 12d ago

You can now convert EXL3 quants on Apple Silicon Mac

Hi, I'm here with an update. But this time it's quite a bigger news on local llm. Normally accessing the high fidelity quant like EXL3 is CUDA gated, and imagine you need 96GB-128GB with RTX cards, they are very specialized and expensive. But now on a more general basis, MacOS…

38
r/LocalLLaMA community 12d ago

z.AI as the number 2 gives praise to the number 1 open source model

  submitted by   /u/Charuru [link]   [comments]

27
r/LocalLLaMA community 12d ago

Any opinion about Qwen3.6-27B@BF16 vs Step3.7@IQ4_XS?

Obviously one is dense, slower but at full precision, and the other is MoE, 7x more params, less than ideal quant, and will eat up more memory, but my question is: Which would make saner decisions with less hand-holding, ie which is genuinely smarter? I've been using…

34
r/LocalLLaMA community 12d ago

Research Project: Injecting Natural-Language Tactical Intent into Multi-Agent Football Policies

Human Intent as a Control Interface for Multi-Agent Systems I've been exploring a project called Football Tactical AI . The idea is simple: Instead of directly controlling players, a human acts as a coach and gives tactical instructions in natural language. For example: "Press…

30
r/LocalLLaMA community 12d ago

[GLM 5.2 UD IQ2_M] That's the best pelican svg image I have ever seen

Computer Specs: rtx 5090 + rtx 3090 (x8 x8 bifurcated) Gigabyte AI TOP B850 Motherboard Ryzen 9950x3d 256gb DDR5 5600 (4x64gb) I didn't have high hopes because of the low quant but damn. This model is capable as hell just by looking at this image I can tell that. The tps is low…

25
r/LocalLLaMA community 12d ago

Best local LLM for English story summarization

Hello, which local LLM is currently the best at story summarization? The stories can be multiple pages long and are in English. Thanks!   submitted by   /u/DesperateGame [link]   [comments]

24
r/LocalLLaMA community 12d ago

Worlds Biggest Chat Title Dataset From SupraLabs

If you search "Chat title dataset" on huggingface a few dys ago, the biggest chat title dataset you would get from it was "ogrnz/chat-titles", but recently at supralabs we have curated a 115K filtered dataset whih breaks the world record for the biggest dataset from 10k samples…

32
r/LocalLLaMA community 12d ago

Best Settings for 48GB VRAM + Qwen 3.6 27B

Hey everyone, I've been running Qwen3.6 27B (Q8_0) across an RTX 4090 + RTX 3090 setup using llama.cpp with tensor split, and I wanted to share what's been working best for me so far. See if anyone has any better settings Hardware: RTX 4090 (24GB) + RTX 3090 (24GB), 48GB VRAM…

4
r/LocalLLaMA community 12d ago

Help with a Local Document RAG System (Storage + Ingestion + Query + Highlighting)

Hey folks, I’m working on designing a local, offline document retrieval + LLM pipeline and would love your input on the architecture. Here’s what I’m aiming for: Storage Upload PDF, DOCX, XLSX, CSV, tables All data stored locally (no cloud) Document Ingestion Watch folder (e.g.,…

29
r/LocalLLaMA community 12d ago

7900XTX 24GB vram, can finally fit Q6K+MTP with Qwen 3.6 27B at 131k context

OS: CatchyOS Instructions: Connect monitor to iGPU directly so when you boot Linux your dGPU vram is 100% free since by default when you use your dGPU it consumes about 700mb~1.2gb of lost context space, yes you can still game normally using this approach. Setup kvcache at…

30
r/LocalLLaMA community 12d ago

GLM 5.2: 98% of max level intelligence with less than half of tokens usage

According to this number of reasoning tokens from GLM 5.1 to GLM 5.2 more than doubled from 16.7k to 36.7k and for me as a local user with old junk Xeon setup this makes GLM 5.2 unusable to the extent where I had to shut down model after 12h of waiting it to respond to my math…

37
r/LocalLLaMA community 12d ago

Some llama.cpp B70 SYCL benchmarks

build: dd4623a74 (9640) | model | size | params | backend | ngl | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: | | gemma4 12B Q8_0 | 11.78 GiB | 11.91 B | SYCL | -1 | pp512 | 1578.19 ± 7.82 |…

11
r/LocalLLaMA community 12d ago

AMD future GPU offerings. Some interesting offerings for a LLM build. What type of LLM rig would you build with these?

https://preview.redd.it/xrgj4u64dd8h1.png?width=1287&format=png&auto=webp&s=2a63e8ea1c6aadf7b61b5820a148a5413a2d7d74 From: https://www.youtube.com/watch?v=U8xBkDBVjPM&t=1459s   submitted by   /u/sooki10 [link]   [comments]

32
r/LocalLLaMA community 12d ago

Local AI for local office files

Which AI agent do you think is the best for working with local files (Excel, PDF, Word, txt, json, etc.)? What have you used for this? What workflows have you implemented?   submitted by   /u/Holiday-Display509 [link]   [comments]

29
r/LocalLLaMA community 12d ago

What is the best book for learning ML/Deep Learning maths?

I am 17 years old and have a particularly deep interest in ai architectures and llms. I regularly read new papers from arXiv and huggingface and tend to understand only half of it, mainly from intuition. However I understand its impossible to understand anything completely…

33
r/LocalLLaMA community 12d ago

Tool calling, opencode qwen3.6 27b 8K

Not sure I'm ready to post an issue in the opencode repo yet but wanted to see if this is common, return to the opencode window after walking away to let it do its thing to find its stopped with this in its thinking.. Started noticing more last week or so, the fix is easy just…

10
r/LocalLLaMA community 12d ago

Giving a local agent web access without paid search/scrape APIs: SearXNG + Scrapling

I wanted web access for a local-first agent without reaching for Tavily, Serper, Firecrawl, etc. For this agent path, I wanted no paid API keys, a search service I control, and page extraction I can run myself. What I ended up with is two tools: web_search and web_extract .…

6
r/LocalLLaMA community 12d ago

Local agent on 4090 - looking for LM Studio settings

I have moved on from Ollama to just dink around and instead want to start running a local agent from time to time. With the 24GB of a 4090 (Gigabyte OC edition) that should be quite possible. But no matter what settings I use for context and batching, token generation is slow as…

36
r/LocalLLaMA community 12d ago

RTX 5090 MSI, only inference or training at 475-500W. Make sure to not bend you cable!

I run this MSI 5090 at 475-500W daily, for mostly diffusion training, or LLM inference. Just by chance I decided to check the cable today and found this. No issues, errors or anything, just all by chance. I never gamed on this card, got it entirely for AI and machine learning.…

8
r/LocalLLaMA community 12d ago

[NEW MODEL] SupraLabs just released supra-title-FFT-preview, 115K samples, almost 10x our first chat title dataset

Hey r/LocalLLaMA ! Following up on Supra-Title-350M-exp (our first chat title generation model), we're releasing supra-title-FFT-preview , trained on a much larger and cleaner dataset. 🤗 supra-title-FFT-preview What changed Our first chat title model was trained on 12K samples…

17
r/LocalLLaMA community 12d ago

I benchmarked Claude's "Fast C++". It wasn't faster

  submitted by   /u/User_Deprecated [link]   [comments]

15
r/LocalLLaMA community 12d ago

Single RTX 3090 (MSI TRio) giving trouble on inference.

Hi, I'm having weird issues with my 3090 on inferencerence via lmstudio , it just: unloads the model/ model crashes + nvidia driver resets freezes the pc gives blue/black screen and the computer restarts or straight up restarts everything. I tried running it regularly,…

33
r/LocalLLaMA community 13d ago

$1800 (in GPU cost running with P2P running Qwen/Qwen3.6-27b-FP8 with 262K context and BF16 KV cache at 55 tok/s

Hey peeps, wanted to share what is possible for folks with an inference only single user use case with 1700 in GPU cost. Setup: 4x 5060 ti (16GB) with P2P If you are in the US and you keep an eye on facebook marketplace and places like slickdeals you can find some 5060 ti 16 GB…

30
r/LocalLLaMA community 13d ago

Best Local Agents - Jun 2026

A megathread that is overdue! Let's discuss and debate on what the best local agents available today are Prologue First a note on terminology: While most regular users are going to have a general sense of what these are, I think its worth a brief pause to preempt turbulence in…

6
r/LocalLLaMA community 13d ago

How do I set the right llama.cpp parameters?

--n-gpu-layers all --ctx-size 0 --reasoning-budget 0 --presence-penalty 1.1 --repeat-penalty 1.1 How do I figure out the optimal llama.cpp parameters for my setup? llama.cpp + Open WebUI in Docker with an AMD GPU (16GB VRAM) running gemma 4 12b and 26b models. Is it all about…

13
r/LocalLLaMA community 13d ago

Maximizing performance of 2x3090 + NVLink

Hey all, I have built myself a decent rig with the following specs: - Ubuntu 24.04 - 2x3090 founder’s with NVLink - Ryzen 7950x3d - 64GB DDR5 I am currently routing my display through an eGPU to maximize available VRAM. My current go-to is Qwen 3.6 27B Q8_0 with MTP and…

6
r/LocalLLaMA community 13d ago

How do you guys setup search with your AI models?

Been selfhosting my models for a while and I'd really like to integrate Gemma 4 12B as a simple voice assistant with search capabilities. I've tried using openwebui but the search is kind of broken with DDG and I really don't want to use API keys from Brave or Google etc. So…

25
r/LocalLLaMA community 13d ago

GLM-5.2-REAP50-GGUF

Has anybody tried these? How do they compare to Qwen 3.6 27b? Model Size Link GLM-5.2-REAP50-Q3_K_M-GGUF 182 GB https://huggingface.co/pipenetwork/GLM-5.2-REAP50-Q3_K_M-GGUF GLM-5.2-REAP50-Q2_K-GGUF 139 GB https://huggingface.co/pipenetwork/GLM-5.2-REAP50-Q2_K-GGUF  …

4
r/LocalLLaMA community 13d ago

Commission selects EUROPA consortium as the winner of the Frontier AI Grande Challenge, a project to build European open-source frontier AI model in all 24 EU languages

The European Commission has selected EUROPA, a European consortium led by the Italian company Domyn, as the winner of its Frontier AI Grand Challenge. Commission selects EUROPA consortium as the winner of the Frontier AI Grande Challenge, a project to build European open-source…

11
r/LocalLLaMA community 13d ago

Improving local models with an API based "consultant"?

I'm sure that someone else has come up with this before, but i just wanted to ask: Has it occurred to anyone to improve their local AI workflow by adding a more powerful API based "consultant" agent (GLM 5.2 now springs to mind) to call upon for refining plans, learnings and…

35
r/LocalLLaMA community 13d ago

The economics of AI are starting to favor open models

For the last couple of years, the assumption was pretty simple: Want the smartest model? Pay for a closed API. Want something cheaper? Accept a capability hit. Looking at recent model releases, that tradeoff is starting to break down. The most interesting part of the chart isn't…

15
r/LocalLLaMA community 13d ago

LQ50-24 English translate

here the full English using google translate for you guy   submitted by   /u/MundanePercentage674 [link]   [comments]

22
r/LocalLLaMA community 13d ago

Best Harness for Web Searching

Looking for opinions on the best software to do web searching resources. What I've tried: LM Studio + plugins Odysseus I think the problem they're both running into is the search engines they're using max out at like, 10 requests per day/hour or something without an api. I don't…

17
r/LocalLLaMA community 13d ago

Benchmarking or benchmarketing?

Maybe I’m getting cynical, but LLM benchmarking is starting to feel less like measurement and more like marketing and positioning. Every week there’s a new leaderboard score, new chart, new eval suite, or some claim that a model is suddenly the best. It feels like benchmarks…

35
r/LocalLLaMA community 13d ago

What's more impressive, GLM 5.1 -> 5.2 or Qwen 3.5 -> 3.6?

Write a single HTML file with a full-page canvas and no libraries. Simulate a realistic Döner Style kebab skewer rotating (vertically) in front of a gas powered heating element. Mentioning Döner activates GLM 5.2s german weights or something (Spiess = Skewer, Brenner = Burner).…

34
r/LocalLLaMA community 13d ago

Is my CPU and RAM too weak/ lees for local LLMs? Both are going 100% for simple test prompts. GPU is not getting used fully. In theory quen3.5:9b should fit and run on RTX3050 8 GB comfortably.

https://preview.redd.it/i69vee9mi88h1.png?width=1592&format=png&auto=webp&s=820720e8a3e1d5386d49119a235e2902acc13265 I am very new to this local llm world. Just started to exploring from past 3days. Share any troubleshooting tips.   submitted by   /u/mr_whoisGAMER [link]…

12
r/LocalLLaMA community 13d ago

Watching a local AI voice assistant get dumber (A 9B to 0.8B agent experiment on my RTX 5060 Ti)

I wanted to find the exact floor for running an intelligent, local voice assistant agent on consumer hardware. I kept the environment, tools, and prompts identical, I stepped the model sizes down through Qwen 3.5 9B, 4B, 2B, and 0.8B to see how agentic reasoning degrades. The…

12
r/LocalLLaMA community 13d ago

GLM-5.2 is the new leading open weights model on the Artificial Analysis Intelligence Index

  submitted by   /u/pscoutou [link]   [comments]

20
r/LocalLLaMA community 13d ago

The Eagle(3) has landed (for Qwen)

https://github.com/ggml-org/llama.cpp/releases/tag/b9723 Available in the latest release. Enabled via: --spec-type draft-eagle3 You'll need to feed it a draft model. There's issues with unsloth + eagle at the moment so I've personally tested against: Model:…

16
r/LocalLLaMA community 13d ago

New Agentic Benchmark Out: Claude Fable and GLM 5.2 Top Their Cohorts

You can read about it here: https://artificialanalysis.ai/articles/aa-briefcase This is a solid benchmark from Artificial Analysis. It basically tests an LLMs ability to plan and execute tasks. And more importantly, it is a new benchmark that is not saturated, so no one can…

32
r/LocalLLaMA community 13d ago

spec: support eagle3 for qwen3.5 & 3.6 by ruixiang63 · Pull Request #24593 · ggml-org/llama.cpp

let's try is it better than MTP   submitted by   /u/jacek2023 [link]   [comments]

5
r/LocalLLaMA community 13d ago

Has anyone here used VibeThinker-3B outside benchmarks?

Just curious, given the hype and benchmark numbers. Curious about real-world behavior: debugging, coding assistance, reasoning over messy prompts, local latency, failure modes, and whether it actually feels useful versus just optimized for verifiable evals.…

23
r/LocalLLaMA community 13d ago

Researchers trained a Deep Research agent with 32 H100s and open-sourced everything

Ohio State University's NLP team released QUEST-35B, an open-source Deep Research agent trained using ~32 H100s and ~8K synthetic samples. The team open-sourced the training recipe, code, weights and datasets. Benchmark results show competitive performance against several…

13
r/LocalLLaMA community 13d ago

EvoTensile: Evolutionary algorithms for AMD Tensile GEMM kernel tuning

There has been an effort to tune kernels in hipBLASLt so the most basic matmuls can run faster. It's known that on Strix Halo (gfx1151), GEMM with NN and TN input layouts (used in inference) are already well-tuned, while NT and TT layouts (used in training) are not yet tuned.…

8
r/LocalLLaMA community 13d ago

GLM-5.2 can now run locally in llama.cpp and Unsloth Studio.

The 2-bit model retains ~82% accuracy after we shrunk it from 1.51TB to 238GB (-84% size). Run on a 256GB Mac or RAM/VRAM setups. GLM-5.2 is the strongest open model to date. Check the graph for the accuracy of each GLM-5.2-GGUF quantization. Full guide:…

35
r/LocalLLaMA community 13d ago

Little late thank you to the DeepSeek team!

7 moths ago I posted https://www.reddit.com/r/LocalLLaMA/s/Z32skdSKzY Just wanted to thank you for DeepSeek V4 Pro and extra big Thank You for the Flash version that fits on my local hardware! Thank You!!!!   submitted by   /u/Sorry_Ad191 [link]   [comments]

37
r/LocalLLaMA community 13d ago

[NEW MODEL] SupraLabs just released SupraVL-Nano-900k, a Vision-Language Model built entirely from scratch!

Hey r/LocalLLaMA ! We just released SupraVL-Nano-900k , our first VLM. It has ~900k parameters, was trained from scratch on Flickr8k, and the entire architecture fits in a single Jupyter notebook. This is not a production model, it's a fully transparent, readable blueprint for…

27
r/LocalLLaMA community 13d ago

SETI @ Home aka distributed LLM inference engine. Does this exist and if not, should we make one?

This seems logical for the benefit of civilisation. I have a 5 GPU system to contribute.   submitted by   /u/HockeyDadNinja [link]   [comments]

26
r/LocalLLaMA community 13d ago

2 weeks since the release of Gemma 4 12b Unified, how are we feeling about it?

I'm looking for a good model to run on a 5090 and have ample context ~128k. This model looks good for me, it seems to have good performance in the 12b range, almost comparable to Gemma 4 26B A4B. Building a custom harness for it and have ~300m of tokens to fine tune on. Do you…

6

Bought 2x r9700, 5090 is now 7k and 6000 pro is at 13.5k, best option for 64 gb vram under 4k

You can now convert EXL3 quants on Apple Silicon Mac

z.AI as the number 2 gives praise to the number 1 open source model

Any opinion about Qwen3.6-27B@BF16 vs Step3.7@IQ4_XS?

Research Project: Injecting Natural-Language Tactical Intent into Multi-Agent Football Policies

[GLM 5.2 UD IQ2_M] That's the best pelican svg image I have ever seen

Best local LLM for English story summarization

Worlds Biggest Chat Title Dataset From SupraLabs

Best Settings for 48GB VRAM + Qwen 3.6 27B

Help with a Local Document RAG System (Storage + Ingestion + Query + Highlighting)

7900XTX 24GB vram, can finally fit Q6K+MTP with Qwen 3.6 27B at 131k context

GLM 5.2: 98% of max level intelligence with less than half of tokens usage

Some llama.cpp B70 SYCL benchmarks

AMD future GPU offerings. Some interesting offerings for a LLM build. What type of LLM rig would you build with these?

Local AI for local office files

What is the best book for learning ML/Deep Learning maths?

Tool calling, opencode qwen3.6 27b 8K

Giving a local agent web access without paid search/scrape APIs: SearXNG + Scrapling

Local agent on 4090 - looking for LM Studio settings

RTX 5090 MSI, only inference or training at 475-500W. Make sure to not bend you cable!

[NEW MODEL] SupraLabs just released supra-title-FFT-preview, 115K samples, almost 10x our first chat title dataset

I benchmarked Claude's "Fast C++". It wasn't faster

Single RTX 3090 (MSI TRio) giving trouble on inference.

$1800 (in GPU cost running with P2P running Qwen/Qwen3.6-27b-FP8 with 262K context and BF16 KV cache at 55 tok/s

Best Local Agents - Jun 2026

How do I set the right llama.cpp parameters?

Maximizing performance of 2x3090 + NVLink

How do you guys setup search with your AI models?

GLM-5.2-REAP50-GGUF

Commission selects EUROPA consortium as the winner of the Frontier AI Grande Challenge, a project to build European open-source frontier AI model in all 24 EU languages

Improving local models with an API based "consultant"?

The economics of AI are starting to favor open models

LQ50-24 English translate

Best Harness for Web Searching

Benchmarking or benchmarketing?

What's more impressive, GLM 5.1 -> 5.2 or Qwen 3.5 -> 3.6?

Is my CPU and RAM too weak/ lees for local LLMs? Both are going 100% for simple test prompts. GPU is not getting used fully. In theory quen3.5:9b should fit and run on RTX3050 8 GB comfortably.

Watching a local AI voice assistant get dumber (A 9B to 0.8B agent experiment on my RTX 5060 Ti)

GLM-5.2 is the new leading open weights model on the Artificial Analysis Intelligence Index

The Eagle(3) has landed (for Qwen)

New Agentic Benchmark Out: Claude Fable and GLM 5.2 Top Their Cohorts

spec: support eagle3 for qwen3.5 & 3.6 by ruixiang63 · Pull Request #24593 · ggml-org/llama.cpp

Has anyone here used VibeThinker-3B outside benchmarks?

Researchers trained a Deep Research agent with 32 H100s and open-sourced everything

EvoTensile: Evolutionary algorithms for AMD Tensile GEMM kernel tuning

GLM-5.2 can now run locally in llama.cpp and Unsloth Studio.

Little late thank you to the DeepSeek team!

[NEW MODEL] SupraLabs just released SupraVL-Nano-900k, a Vision-Language Model built entirely from scratch!

SETI @ Home aka distributed LLM inference engine. Does this exist and if not, should we make one?

2 weeks since the release of Gemma 4 12b Unified, how are we feeling about it?