r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 9d ago

DeepSeek-V4-Flash-Vision-Exp

  submitted by   /u/Xhehab_ [link]   [comments]

8
r/LocalLLaMA community 9d ago

Fastest NVFP4 quant of Qwen3.8 27B out there

Here's a brand new Blackwell-native, prefill-optimized 4-bit quant that runs 50% faster on compatible hardware than a Q4 quant of the same memory footprint. And it runs 4-7% faster than other NVFP4 quants as benchmarked on RTX 5090 32GB. Quant Benchmark Speed NVFP4 pp2048 6250…

27
r/LocalLLaMA community 9d ago

Does PCIe matter much for inference?

Speaking of -sm tensor. I have 2x5060ti and i get some 40-50tps form 3.8 27B q6. I used HWinfo to see how saturated the PCIes are during inference are and as expected both the PCIe 5x16 slot and PCIe 4x4 were fully saturated. I cant help but feel like my 2nd gpu slot is a…

7
r/LocalLLaMA community 9d ago

New stealth model on OpenRouter: Ox Alph

Any idea which lab this is from? people are guessing this is a Chinese model.   submitted by   /u/Neosinic [link]   [comments]

36
r/LocalLLaMA community 9d ago

Useful > Fast

I run 2 x 3090 on a ryzen 7 with 32gb ddr4 6000. I see a lot of posts about maxing speed / context pool. I’ve done this myself with 3.8 27b. What I’m interested in though is once the dust settles and we look at utility, what balance people are striking in terms of similar setups…

5
r/LocalLLaMA community 9d ago

Qwen3.8-27B at 262K context on a Strix Halo + RTX 3090 Ti: 9.5 -> 153 tok/s, and it beats a dual-3090 vLLM box on HumanEval

Spent a while treating layer placement, KV format and llama.cpp itself as experimental variables. 159 logged experiments. Numbers first, caveats after. Hardware: AMD Ryzen AI MAX+ 395 (Strix Halo, 128 GB unified) + RTX 3090 Ti on an eGPU link. One llama.cpp process, AMD on…

37
r/LocalLLaMA community 9d ago

Fastest qwen 3.8 27b for AMD gpu?

Hey, just wondering if there are forks or exact gguf versions that give fastest prompt processing and token gen speeds for AMD gpu? Looking to run q8 or q6 Vram 96gb W7900 + w7800 both 48gb With bandwidth mismatch, tensor paralleling amd equivalent not working   submitted by…

21
r/LocalLLaMA community 9d ago

Ox Alpha stealth model: GLM5 Air, Mimo V3 or ?

To anyone who needs AIR…   submitted by   /u/Miserable-Dare5090 [link]   [comments]

27
r/LocalLLaMA community 9d ago

New stealth model Ox Alpha look like glm or mimo new model

https://preview.redd.it/stgzmbpgnnkh1.jpg?width=2179&format=pjpg&auto=webp&s=19a4fd0e1786cbcac0967e1bf1ec4898447e989a https://preview.redd.it/paub9gylnnkh1.png?width=680&format=png&auto=webp&s=c13efb53356b1b7e7826aece96914cad9252240c Ox Alpha is a frontier model built for…

30
r/LocalLLaMA community 9d ago

SenseNova U1.5-Lite full release: expert training, OPD distillation, one model at inference

Benchmarks: Benchmark U1 Preview Full Qwen-Image-Bench 47.14 55.20 (PE) 60.18 (PE) ImgEdit 3.9 4.37 4.59 GEdit-Bench-EN 7.47 8.14 8.26 Instead of just scaling up, they train task-specialized expert models for text rendering and infographics, aesthetic quality, and image editing.…

15
r/LocalLLaMA community 9d ago

Is there any interest in a self hosted open source version of manus/perplexity?

I have been working on a project for about 3 months because I found Vane to be awful and I wanted something like Manus for myself. I have a pretty solid app that's probably about ready to release, but I'm not sure if there is any desire there. It does research at different…

9
r/LocalLLaMA community 9d ago

Make Jensen Huang Sound Like Anyone. New Streaming Voice Conversion Model MeanVC2 Released!

Finally see a new voice conversion model. MeanVC2 supports cross-gender and cross-language voice conversion. 3x realtime on CPU with audio.cpp. Disclaimer: The converted voice quality of MeanVC2 is decent; the noise comes from my rough demo engineering, not the model itself.…

38
r/LocalLLaMA community 9d ago

I did it! I'm free! It's been 7 hours since I used claudecode

My Pro subscription expired today, they killed my access at 1pm local time. I'm now using Qwen3.8-27b w/ 5090m 24gb vram and pi to do everything i was doing in claudecode. The only downside is claudecode let me code without using my gpu, meaning I have to plan things now. Last…

15
r/LocalLLaMA community 9d ago

Qwen 3.8 27b - PI AGENT vs OPENCODE

https://www.reddit.com/r/LocalLLaMA/comments/1j7r47l/i_just_made_an_animation_of_a_ball_bouncing/ This post inspired me to make that test after a year ;) That is one of my many tests I make comparing output quality. What is more interesting using a PI Agent results are much…

35
r/LocalLLaMA community 9d ago

Gonna be huge for US open source

  submitted by   /u/pmv143 [link]   [comments]

32
r/LocalLLaMA community 9d ago

Been tweaking my Qwen 3.8 setup, up to 45+ steady T/ps at 8bit quant. Realised I'm now top T/ps for this model+ctx across all benchmarked M-series chips. Full args linked below, happy to discuss as this was a pain of trial and error.

https://omlx.ai/benchmarks/performance/2pko3m1k - you can expand the raw args, but I have full annotations of what worked and what didn't. I'm now testing the model on acutal coding and haven't seen any issues with performance vs default suggested vals for the vanilla model.…

35
r/LocalLLaMA community 9d ago

DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretation

Their summary: For over a decade, we’ve accepted that end-to-end backprop is the only way to train deep networks. But holding the entire network in memory all at once is why AI training is hitting a resource wall. . We found a new way to break the network into blocks and train…

31
r/LocalLLaMA community 9d ago

A happy user of llama-swap

I recently considered purchasing another GPU, not so much because I need more bandwidth or vram, but because I have several local GPU-based scheduled workloads that use different models/config. After shopping around looking at prices, I decided to use a different approach. After…

29
r/LocalLLaMA community 9d ago

Fine-tuning Cactus Needle 2 can match DeepSeek v4 on the specific task

Hey LocalLlama, Henry from Cactus here! When we trained Needle 2, I had a strict rule to not expose the model to any data sample that remotely felt like these benchmarks. It seemed over-the-top, but benchmarks are easy to overfit around, yet struggle in the wild, especially…

9
r/LocalLLaMA community 9d ago

I pushed Qwen3.8-27B to 381 tps for a single request on a RTX 3090

Four days ago I released a hyper-optimized Qwen3.8-27B inference engine for an RTX 3090 (82 tps single request, 672 peak). Since then it went to ~114, then ~138 tps single-user with DFlash2 drafting and lookup-augmented drafting. Today it's ~133 tps on real chat prompts, 382 tps…

15
r/LocalLLaMA community 9d ago

If you are wondering why Ornith 1.5 35B A3B with MTP is so slow, this is why

The author of the HuggingFace post discovered that Ornith-1.5-35B-A3B is currently being shipped with a MTP head that was never actually trained — it's just random initialization.   submitted by   /u/Max-_-Power [link]   [comments]

21
r/LocalLLaMA community 9d ago

NVIDIA dropped an NVIDIA-hosted CUDA MCP for AI-assisted CUDA operations, such as searching official, up-to-date documentation, writing optimized GPU code, and analyzing performance data

  submitted by   /u/swagonflyyyy [link]   [comments]

31
r/LocalLLaMA community 9d ago

Any speculation on whether or not Google will announce a new Gemma model at the Gemma SF Celebration tonight?

From the Digg article ( https://digg.com/tech/3pf3046j ) “Google Gemma posted that the family of open models has achieved 1 billion downloads. The account is hosting an exclusive evening in San Francisco on August 20 to honor open-source builders, researchers, and contributors.…

12
r/LocalLLaMA community 9d ago

Qwen3.8-27B scored 29/30 on AIME 2026 with FP8 + xhigh reasoning — BF16 vs FP8 results

I benchmarked Qwen3.8-27B on MathArena/aime_2026 dataset, comparing BF16 and FP8 weights at medium and xhigh reasoning effort. Interesting findings are: quantized FP8 xhigh is better than BF 16 medium equally good as 16 BF xhigh with better speed. On problem 7, both BF16 xhigh…

28
r/LocalLLaMA community 9d ago

AQuA's "self-improvement" updates research state, not the agent LM. What should a local port freeze?

AQuA's preprint uses "recursive self-improvement" for a bounded research loop. It does not say the research-agent LM rewrites its own weights. The paper separates three objects: The language model driving the research agent and the evaluator stay fixed within each part. Each…

6
r/LocalLLaMA community 9d ago

Unsloth Dynamic 3.0 GGUFs

  submitted by   /u/kexxty [link]   [comments]

12
r/LocalLLaMA community 9d ago

Ladies and gentlemen I present to you Qwen3.8 27b 1bit brain damage quant

I wanted to just test the unsloth 1bit quant of qwen 3.8 27b as I have just 8gb vram and ngl it gave me a good laugh   submitted by   /u/Ok-Health-7096 [link]   [comments]

6
r/LocalLLaMA community 10d ago

Theres surely SOMEONE out there whose job is just pumping out low-poly oneshot ThreeJS assets..

..for that person AGI has effectively been achieved and is fully local. For everyone else - come up with new tests please!! I'm going to lose my mind if I see another tech influencer post that damn pagoda   submitted by   /u/ForsookComparison [link]   [comments]

6
r/LocalLLaMA community 10d ago

Ling-3.0 released all 6 base checkpoints: 2 sizes × 3 stages

AntLing has released the full six-checkpoint matrix for the Ling-3.0 base model. tiny: pretrained, mid-trained, WSM-merged flash: pretrained, mid-trained, WSM-merged The concrete artifact is six separate official repositories, not one endpoint repeated under different names. All…

32
r/LocalLLaMA community 10d ago

QwenMix-3.7: Kept seeing posts about Qwen3.8 and 3.6 sharing the same structure.. so I had Qwen3.8 combine them.

I chose to do this thing, not because it was hard, but because it was silly. Posts kept discussing how 3.8 and 3.6 were functionally the same, but based on training (3.8 does have seven new tokens!).. so I figured I'd see if they could be merged. They can. I used…

34
r/LocalLLaMA community 10d ago

Getting better at coding doesn't make a model better at everything else

A majority of users in this sub use LLMs for coding/agentic tasks and I see why a lot of value is put into them but many try to say "Well coding has improved therefore it can just use tool calling and/or just look up what the user needs if there's a degradation for general…

37
r/LocalLLaMA community 10d ago

[MASSIVE TINY RELEASE] - Supra2-Medium-Base - a tiny 25M parameters model competing heavily with our previous 50M model!

Hey guys! Supra2-Medium is finally out! It's a 25M parameters qwen3 architecture model trained entirely from scratch (on our new rig: RTX 5060 Ti 16GB + the new RTX 5060 8GB!). Here's how it competes in benchmarks with Supra-50M-Base (which is double as large!!):…

8
r/LocalLLaMA community 10d ago

Qwen3.8 27b just exceeded my expectations on svg generation :D

https://reddit.com/link/1vtkgdj/video/595yn0ckdjkh1/player I wanted to try out Qwen3.8 27B 's SVG capabilities but with something different than the pelican on a bicycle. Promt was literally just : create a single html file with an embedded svg of a cat riding a zebra, riding an…

9
r/LocalLLaMA community 10d ago

The boring way to run Deepseek V4 Flash-0731 130-150 tks - 16x5060ti 16GB over 2 PLX88096 switches

Component Validated configuration Motherboard ASRock Rack SPC621D8U-2T/OVH CPU Xeon Gold 6330 (Get gold/platinum if interested in Optane Pmem gimmicks) GPU fabric Two Broadcom/PLX PEX88096 islands, eight GPUs per island GPUs 16 x RTX 5060 Ti 16 GB OS Ubuntu 22.04.5 LTS Kernel…

15
r/LocalLLaMA community 10d ago

Aurora-80K releases! A modern tiny language model.

I'm introducing Aurora-80K, a small language model with exactly 80 thousand parameters. It uses a factorized 4,096-token vocabulary despite having only 80K parameters. The benchmarks: Wikitext-2 BPB: 3.2902 BLiMP: 52.31% Arc-Easy: 26.05% More information about the model is…

7
r/LocalLLaMA community 10d ago

Tencent begins testing its new flagship model Hunyuan Hy4

From the screenshots: Hy4 is now live, labeled "Expert-Level Model" + "Use Tools to Solve Problems" Hy3 is tagged with "New Upgrade," positioned as a brand-new general-purpose model DeepSeek, focused on reasoning, is listed alongside it From SuSu_酥酥👅on 𝕏:…

18
r/LocalLLaMA community 10d ago

I just built a mini Kimi-K3 from Scratch under 250$. Already beats GPT-2 (124M)!

I pre-trained a 1.02-billion-parameter on Kimi K3 replica trained on 5.00 billion decontaminated tokens for $250. This model has 1.02 billion parameters, of which 145 million are active per token. It is roughly one two-thousandth of K3 by total size. It saw 5,000,003,584 tokens,…

12
r/LocalLLaMA community 10d ago

[Draft - Open PR] AVX2: Speed up large batch size prompt processing of IQ models by bartowski1182 · Pull Request #27402 · ggml-org/llama.cpp

IQ quants are particularly slow on CPU at large batch sizes (what you'd see for imatrix and perplexity) Benchmark numbers I ran PPL against master and this PR to get speed and numbers on --chunks 50 for Qwen3.6-27B and Qwen3.6-35B-A3B on EPYC 9654 using 24 threads Created pure…

23
r/LocalLLaMA community 10d ago

AirLLM - Recent Updates - with Qwen3.8-27B, Kimi-K3 too

AirLLM dramatically reduces inference memory usage, letting 70B large language models run on a single 4GB GPU card — without quantization, distillation, or pruning. You can even run 405B Llama 3.1 on 8GB , DeepSeek-V3 (671B) on ~12GB , and Kimi K3 (2.8T) — the largest…

12
r/LocalLLaMA community 10d ago

G9v3-39A5B on artificialanalysis looks good. Has anyone tested it?

I see https://github.com/linuxid10t/llama.cpp/tree/feature/g9v3-support but yeah... Considering that some results place it above Qwen 3.6 27B (the top performer until a few days ago) and that it is an MoE model, I think it could be interesting.…

22
r/LocalLLaMA community 10d ago

New benchmark just dropped!

The pelican on a bicycle is sooo outdated, so I came up with a new, improved version. Qwen3.8-27b medium (UD-Q4_K_XL) vs. Sol 5.6 high vs. Qwen3.6-35B (UD-Q6_K_XL) Prompt (only real with typo!): "Create a svg of a horse on a blue bycicle in the desert, with a camel in the…

15
r/LocalLLaMA community 10d ago

TinySearch v0.6.1 - still a lightweight web research tool for local LLMs, now with bring-your-own-browser support

Hey everyone, Posted TinySearch here a few versions ago and got a bunch of useful feedback, so figured I'd post an update because the thing has changed quite a bit since then. Repo: [ https://github.com/TinySuiteHQ/TinySearch]() The basic idea is still the same: TinySearch is a…

30
r/LocalLLaMA community 10d ago

Qwen 3.8 27B KV f16 vs q8_0 are not equivalents

I'm testing it since release, now with UD 3.0 in my AMD R9700 with ROCm, I always read everywhere that F16 and q8_0 for KV cache are essentially the same... well, I tested it and I can see differences. Some differences are minimal, F16 is more careful and detailed with…

9
r/LocalLLaMA community 10d ago

3 days benchmarking most llama.cpp flags on my weird 40gb vram laptop + tb4 egpu setup. Got +70% generation, +40% prefill, 60k more context, and filed a bug in llama around MTP. What I learned.

tldr: went from 16~ t/s to 27~ t/s generation. got my usable context up from 220k to the full 262k without sacrificing anything. prefill also increased from 376 to 573 command I ended up with, fwiw: llama-server -m Qwen3.8-27B-UD-Q6_K_XL.gguf -c 262144 -ngl 999 -fa on \ -ctk…

5
r/LocalLLaMA community 10d ago

Spider-man: Brand New Day, does Peter self host his AI? (Spoilers)

This is gonna seem crazy off-topic, but I saw Spider-Man the other day and couldn’t help but notice how well executed E.V. is as an agentic system. compared to Jarvis, it felt much more grounded and closer to reality. No wild holographic interface, just a guy talking to his…

37
r/LocalLLaMA community 10d ago

Qwen3.8-27B took a serious hit to *knowledge* vs 3.6

Like many of you I've spent the last few days throwing Qwen3.8-27B against all of my usual use-cases and personal tasks/harnesses and workflows. It's great, phenomenal sometimes, but that's not what this post is about. One of my little personal benchmarks is a little set of…

10
r/LocalLLaMA community 10d ago

Qwen3.8-27b has the highest level of "agency" I've ever seen in a local model

Off a single prompt, given my credentials and the name of my university, qwen3.8-27b was able to successfully pull my class schedule from the kinda shitty and convoluted web of university websites. It needed no human intervention, and executed 80 tool calls. Another time, I…

28
r/LocalLLaMA community 10d ago

Qwen3.8-23B-Mini-Me: A Depth-Pruned Qwen3.8-27B (to ~22.7BB)

I've been working on a depth pruning approach and decided to try it out on the new Qwen3.8-27B model. I managed to get the model down to about 22.7B params without severe reasoning degradation. No fine-tuning was done, just strategic removal of layers. It's been working well for…

13
r/LocalLLaMA community 10d ago

Qwen 3.8 27B SlopCodeBench results

Howdy, I'm back again - running my favorite benchmark (it's still unsaturated for the time being so might as well!) previous runs a b https://github.com/michaelasper/benchmarks/blob/main/qwen3.8-27b-pi-on-slop-code-bench.md I ran this via OpenRouter because my mac would cry…

10
r/LocalLLaMA community 10d ago

Anyone NOT on full auto when coding with local LLMs?

Would love to know who's letting a 9B just go ham locally, haha But in all seriousness, how many of you are keeping to manual or manual-ish dev workflows?   submitted by   /u/BatPlack [link]   [comments]

23

DeepSeek-V4-Flash-Vision-Exp

Fastest NVFP4 quant of Qwen3.8 27B out there

Does PCIe matter much for inference?

New stealth model on OpenRouter: Ox Alph

Useful > Fast

Qwen3.8-27B at 262K context on a Strix Halo + RTX 3090 Ti: 9.5 -> 153 tok/s, and it beats a dual-3090 vLLM box on HumanEval

Fastest qwen 3.8 27b for AMD gpu?

Ox Alpha stealth model: GLM5 Air, Mimo V3 or ?

New stealth model Ox Alpha look like glm or mimo new model

SenseNova U1.5-Lite full release: expert training, OPD distillation, one model at inference

Is there any interest in a self hosted open source version of manus/perplexity?

Make Jensen Huang Sound Like Anyone. New Streaming Voice Conversion Model MeanVC2 Released!

I did it! I'm free! It's been 7 hours since I used claudecode

Qwen 3.8 27b - PI AGENT vs OPENCODE

Gonna be huge for US open source

Been tweaking my Qwen 3.8 setup, up to 45+ steady T/ps at 8bit quant. Realised I'm now top T/ps for this model+ctx across all benchmarked M-series chips. Full args linked below, happy to discuss as this was a pain of trial and error.

DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretation

A happy user of llama-swap

Fine-tuning Cactus Needle 2 can match DeepSeek v4 on the specific task

I pushed Qwen3.8-27B to 381 tps for a single request on a RTX 3090

If you are wondering why Ornith 1.5 35B A3B with MTP is so slow, this is why

NVIDIA dropped an NVIDIA-hosted CUDA MCP for AI-assisted CUDA operations, such as searching official, up-to-date documentation, writing optimized GPU code, and analyzing performance data

Any speculation on whether or not Google will announce a new Gemma model at the Gemma SF Celebration tonight?

Qwen3.8-27B scored 29/30 on AIME 2026 with FP8 + xhigh reasoning — BF16 vs FP8 results

AQuA's "self-improvement" updates research state, not the agent LM. What should a local port freeze?

Unsloth Dynamic 3.0 GGUFs

Ladies and gentlemen I present to you Qwen3.8 27b 1bit brain damage quant

Theres surely SOMEONE out there whose job is just pumping out low-poly oneshot ThreeJS assets..

Ling-3.0 released all 6 base checkpoints: 2 sizes × 3 stages

QwenMix-3.7: Kept seeing posts about Qwen3.8 and 3.6 sharing the same structure.. so I had Qwen3.8 combine them.

Getting better at coding doesn't make a model better at everything else

[MASSIVE TINY RELEASE] - Supra2-Medium-Base - a tiny 25M parameters model competing heavily with our previous 50M model!

Qwen3.8 27b just exceeded my expectations on svg generation :D

The boring way to run Deepseek V4 Flash-0731 130-150 tks - 16x5060ti 16GB over 2 PLX88096 switches

Aurora-80K releases! A modern tiny language model.

Tencent begins testing its new flagship model Hunyuan Hy4

I just built a mini Kimi-K3 from Scratch under 250$. Already beats GPT-2 (124M)!

[Draft - Open PR] AVX2: Speed up large batch size prompt processing of IQ models by bartowski1182 · Pull Request #27402 · ggml-org/llama.cpp

AirLLM - Recent Updates - with Qwen3.8-27B, Kimi-K3 too

G9v3-39A5B on artificialanalysis looks good. Has anyone tested it?

New benchmark just dropped!

TinySearch v0.6.1 - still a lightweight web research tool for local LLMs, now with bring-your-own-browser support

Qwen 3.8 27B KV f16 vs q8_0 are not equivalents

3 days benchmarking most llama.cpp flags on my weird 40gb vram laptop + tb4 egpu setup. Got +70% generation, +40% prefill, 60k more context, and filed a bug in llama around MTP. What I learned.

Spider-man: Brand New Day, does Peter self host his AI? (Spoilers)

Qwen3.8-27B took a serious hit to *knowledge* vs 3.6

Qwen3.8-27b has the highest level of "agency" I've ever seen in a local model

Qwen3.8-23B-Mini-Me: A Depth-Pruned Qwen3.8-27B (to ~22.7BB)

Qwen 3.8 27B SlopCodeBench results

Anyone NOT on full auto when coding with local LLMs?

Qwen3.8-27B took a serious hit to knowledge vs 3.6