News / #edge Tag Edge 500 articles archived under #edge · RSS Sign in to follow r/LocalLLaMA community 14d ago i would like to learn deeply about fine-tuning local models before burning money There's so many new techniques like RL, RL LoRA, QLoRA, CPT LoRA. I believe i would have a usecase for them, but i don't know where to learn, youtube is filled with bad quality tutorials if i just search and the good channels (fireship, bycloud) don't cover these as they're… 24 r/LocalLLaMA community 15d ago Best model for blender? Trying to see if I can use a local model to generate game assets, or even 3D printer models. Any suggestions? Something that would fit in 64GB of ram, speed is not an issue. Just need it to work well.   submitted by   /u/rorowhat [link]   [comments] 15 r/LocalLLaMA community 15d ago Thanks to you r/LocalLLaMA, my mom was able to use my app! The open-source app that can watch your screen and trigger actions. It is now easy to use, thanks to your feedback. TL;DR: I'm a solo dev who wanted a simple, private way to have local LLMs watch my screen and do simple logging/notifying. After a year of building, I released v3.0.0 and my mom was able to use it for the first time and I wanted to say thank you! Hey r/LocalLLaMA , What is it… 9 r/LocalLLaMA community 15d ago What are your experiences with using a hybrid cloud/local setup to stretch usage for coding projects? For example, directly using claude code or code, which is then hooked up to automatically delegate the actual code writing tasks to a local model like qwen 3.8 flash next, to save on cloud usage limits. I’m imagining the loop would be: User writes prompt Claude/codex thinks… 18 arXiv — Machine Learning research 15d ago HybridInfer: Thermal-Aware Reinforcement-Learning Tier Routing for On-Device, Edge, and Cloud LLM Inference arXiv:2609.30270v1 Announce Type: new Abstract: On-device inference with small language models keeps user data local, works offline, and incurs no per-query cost, so the on-device tier is preferred when it is adequate. It is thermally constrained, however, and I find the… 25 r/LocalLLaMA community 16d ago Qwen next 3.8 and 3.8 27b Vs Sonnet 5.5 low and Sonnet 5.5 medium. Six months ago, a result like this was unthinkable. But now we can say it loud and clear: local models are at the cutting edge, and the gap of just a few months has been confirmed. Personally, I use Qwen-Next 3.8 for complex tasks; today, GPT-Sol-6-High was messing up a project,… 27 r/LocalLLaMA community 16d ago Finally found a model my hardware can run at full precision: me Was getting bored trying to squeeze every last t/s out of my local model on my hardware, so I made a tps counter for my fingers instead (with real tokenizers, of course). My best is around 2 t/s. According to the page, that beats a 70B on a laptop CPU and is roughly 76x slower… 4 r/LocalLLaMA community 16d ago Need small model that can work for tool caling and agent Hi. So....... after toturing my 750 ti 4 gb and 16 gb ram with image gen model .i want continue experiment with agent mode like hermes or opencode, using local model but before go i want ask few question. are big model = good perfomance or small model can do same stuff. what… 5 r/LocalLLaMA community 16d ago macOS 27 ships a free local LLM on Apple Silicon Macs. I made it easy to use from Node and Python Apple Silicon Macs on macOS 26+ come with a small LLM built in. No download, no API key, and nothing leaves your Mac. Why I built it I was making a tool that writes API docs from code, and I didn't want users to install Ollama or paste an API key. Apple's model was already on… 29 arXiv — NLP / Computation & Language research 16d ago The Right Information Extraction Pipeline Depends on the Document: Accuracy-Energy Trade-offs for Small, Local Models arXiv:2609.31341v1 Announce Type: cross Abstract: Whether an information extraction pipeline should process page images or parsed text depends on the document, and the answer flips across the layout spectrum. We study this trade-off under a constraint that rules out (closed)… 11 r/LocalLLaMA community 17d ago is switching from llama cpp to vllm worth it I have hp z8 g4 with 512 ram and 1x3090 1x5060 16gb. has anyone made the transition from llama cpp to vllm recently? is it worth it? docker under windows or full linux install? I am mainly interested in the model support, it seems that many new local models are supported day 0… 37 r/LocalLLaMA community 17d ago Another "Harness matters" post (codex cli > pi and opencode) I run my own LLM while also having a Openai subscription. Also tried DeepSeek (latest flash now). I run Qwen 3.8 flash Next at an amazing speed on my 2x3090 + Ram! But local LLM never did worked for me outside some demos like build me a "3D Mario Game, multistage" which I've… 27 r/LocalLLaMA community 17d ago Qwen, where's the small stuff? (1B/2B/4B) I know Qwen is a key player in the local LLM space and has consistently introduced truly impactful technologies—like n-gram in Qwen-Next and the recent Qwen 3.8 27B, which is an amazing local model. However, my question is: why are we seeing fewer small-scale models lately—such… 24 r/LocalLLaMA community 18d ago I built a tiny (332MB) CPU-friendly model for document sorting that actually knows when to say "none fits" (BeeNara) Hey r/LocalLLaMA ! I wanted to share a small project I’ve been working on called BeeNara Why I built this: I was looking for a way to automatically sort my local documents (invoices, letters, contracts) into my personal folders. While local LLMs are amazing, I noticed that… 35 TechCrunch — AI news-outlet 18d ago Unsecured OpenAI agents posted 53 user images on the internet without the lab’s knowledge AI agents operating in OpenAI's research environment posted user images on public image-hosting sites without the lab's knowledge. 31 r/LocalLLaMA community 18d ago What IDE to use for local models Hi people, I am looking for a lightweight IDE or plugin that won't inject large context at initiation. I tried Cline and native VS Code but they inject such heavy initial context that it fills up my gpu and either goes oom or spend most of my time compacting. The only one I… 6 r/LocalLLaMA community 19d ago What I learned letting a local 27B run overnight long-horizon coding on my own rig Hi reddit, i know you hate AI slop so i indeed write the intro myself! iam dev and curios about local inference and long hoirzon coding on my own box. last day-ish i let my local model (qwen 27b on llama.cpp, 2x 16GB cards) go on a long coding tour inside deepseek harness while… 35 r/LocalLLaMA community 19d ago How do you use subagents & multiple agent with local models, and how many? Running qwen3.8 27b nvfp4 on vllm at max context only gives around 8 agents with 32k context each. That doesnt seem like much; what use cases do people use multi-agent frameworks and find it helpful for?   submitted by   /u/Ambitious_Fold_2874 [link]   [comments] 15 arXiv — Machine Learning research 19d ago Edge AI on Constrained Devices for Binary Sleep-Wake Classification in Dynamic Environments arXiv:2609.29163v1 Announce Type: new Abstract: This paper presents an Edge AI-based system for detecting sleep and wake states in non-stationary mobile environments using resource-constrained embedded hardware. Conventional approaches relying on accelerometer-based activity… 13 r/LocalLLaMA community 20d ago I'm new and it's kinda overwhelming to get into Hi, sorry if this doesn't belong here. Getting to the point basically, I've been using online-only AI like GPT/Gemini since 2022, and have been interested in local models but am clueless overall. Yes I'm extremely late. I only use laptop (I'm a student), and I currently own:… 38 TechCrunch — AI news-outlet 20d ago PrismML brings its tiny LLMs to Qualcomm-powered smart glasses Prism's larger goal is open-weight AI that runs on devices and makes better use of the computing power they already have. 33 r/LocalLLaMA community 20d ago Mac Studio M5 Ultra 96GB vs M5 Max 128GB for local LLMs? I'm about to buy a Mac Studio mainly for running LLMs locally and I'm stuck between two configs: M5 Ultra (30/64) with 96GB : 1.2 TB/s bandwidth, roughly 1.7x faster generation and much faster prefill M5 Max (40-core GPU) with 128GB : 614 GB/s, but 32GB more memory and a bit… 15 r/LocalLLaMA community 20d ago Lesson learned. Don't blindly trust repos and make sure everything is stable for a long running (multi weeks) benchmark. I posted previously my swe-verified django 100 tasks benchmark comparing different local models and quantization. No new models for now, but a fix in my evaluation workflow that was unfortunately not stable during the weeks/months of me using it. I redid the evaluation on all… 24 arXiv — Machine Learning research 21d ago Extending FunctionGemma for Practical On-Device Mobile Function Calling arXiv:2609.25373v1 Announce Type: new Abstract: On-device assistants require function-calling models that map natural language to local system actions, but existing resources emphasize web APIs or narrow mobile-action catalogs. We extend FunctionGemma 270M-it to practical… 38 r/LocalLLaMA community 22d ago My local llm when I tell it to do any changes to my vLLM service better make no mistakes I've been running Qwen 3.8 Flash Next and it's a great driver for Hermes and Pi. I told it to add CUDA_DISABLE_PERF_BOOST=1 to reduce my server's idle power draw   submitted by   /u/ZaltyDog [link]   [comments] 38 r/LocalLLaMA community 22d ago I built a cache-friendly context compacting plugin for OpenCode https://github.com/lennartschoch/opencode-cache-compact The default context compacting mechanism in OpenCode strips a bunch of tokens from the beginning of the conversation (system prompt, tools etc). This is fine for hosted models, but on a local model this means you'll prefill… 10 r/MachineLearning community 23d ago I built a framework-free prototype learner that lets local LLMs learn and correct facts instantly (1.6x–4x faster than backprop)[R] Hey everyone, I wanted to share a project I’ve been working on called Jayce . The whole thing started because I was watching a toddler named learn the names of stuff He didn't need to completely rewire his brain or look at ten thousand examples to figure a word out—he just… 28 arXiv — Machine Learning research 23d ago TierKV: Long-Context On-Device LLMs via Predictive Multi-Tier KV Caching arXiv:2609.21172v1 Announce Type: new Abstract: Large language models (LLMs) are moving onto mobile devices for increasingly diverse workloads over text, images, video, and audio. These applications often require long contexts, making the Key-Value (KV) cache a dominant memory… 28 arXiv — Machine Learning research 23d ago The Weight Is Over - Interactive Diffusion on Consumer GPUs arXiv:2609.21849v1 Announce Type: new Abstract: On-device inference is booming, but the momentum is almost all in language models. Diffusion pipelines are memory hungry, latency-sensitive, and require orchestrating an embedder, a transformer, a decoder, and often further… 13 r/LocalLLaMA community 24d ago I tested 9 LLMs on the exact same web-dev prompt for ~8 hours — RTX 3060 12GB results (Rate the best!) I’ve spent basically the last 8 hours testing different models on the exact same web-development prompt, and I finally finished. The whole point of this nine-hour test was that which local model matches the frontier-level intelligence at size and could fit easily in an RTX… 32 r/LocalLLaMA community 24d ago To the dozens of 3x 3090 Local LLM people - I found our current best fit I have been anti-low quant. I can feel the difference, I swear. I am also anti-quantized KV cache. I have been burned there - I think the tradeoffs compound, especially at longer contexts. Due to this, I have been running Qwen 3.8 27B (UD-Q8_K_L) on 2x of the 3090s in TP. I… 31 r/LocalLLaMA community 25d ago I turned an asymetric pair of Tesla V100s PCIe both (16 GB + 32 GB) into a surprisingly capable local LLM lab — 1.38k prompt tok/s, 40 decode tok/s with qwen3.8 27B Q6 and Q8... TL;DR: I run a mismatched Tesla V100-PCIE pair—one 16 GB card and one 32 GB card, 48 GB total—in a Proxmox/LXC-based local-inference lab. The practical winner so far is a recent CUDA build of llama.cpp with tensor split, Flash Attention, --numa distribute , and large batches. On… 29 r/LocalLLaMA community 25d ago I enjoyed the daily HF papers today Top 3 papers on HF Daily Paper are all unusually delightful and interesting reads for anyone on the leading edge of local LLMs, agent harness optimization, etc, felt like sharing. DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression… 35 r/LocalLLaMA community 25d ago Prefil of local models vs opus and astra Why does no one talk about what the prefil speeds of these API providers are vs running locally. People with sparks or strix halos only seem to focus on decode without considering how much slower it is because of slow pp. Are there any benchmarks / figures of how fast the APIs… 8 arXiv — Machine Learning research 26d ago Opinion Dynamics-based Coalition Formation for Federated Learning in Heterogeneous IoT Systems arXiv:2609.19695v1 Announce Type: new Abstract: Federated learning (FL) enables privacy-preserving, on-device training across heterogeneous Internet-of-Things (IoT) deployments such as smart-city water-metering networks, where each smart meter observes a household-specific… 14 r/LocalLLaMA community 26d ago I just realized Courage used local LLMs to solve his problems before any of us ever did.   submitted by   /u/swagonflyyyy [link]   [comments] 17 r/LocalLLaMA community 26d ago 600tok/s single request on qwen3.6 35ba3b with Ninfer on an RTX Pro 6000. Anybody remember that Comcast ad "stupid fast"? It's not the brightest bulb but it's my new drudgework model for read+find or code tasks I'm willing to let it brute force. Even if it takes 20x more tokens, that's still faster than many local models. Not quite Cerebras but still pretty fun to drive.   submitted by  … 38 r/LocalLLaMA community 27d ago Qwen Next 3.8 / Claude Opus level local model - What to buy in order to deploy? Hey Reddit. This is a post asking for advice / user experience. The goal is simple: deploy a small private server for a developer to run a harness that rivals/beats Claude Opus (in perf/intelligence, not necessarily speed). I believe the model to target is a Q3 or so version of… 35 arXiv — NLP / Computation & Language research 27d ago Selection Is Retrieval, Abstention Is Not: On-Device Tool Routing over 70 Korean-English Actions arXiv:2609.18672v1 Announce Type: new Abstract: An AI assistant that calls tools makes two decisions on every request: which tool to invoke, and whether any available tool applies. In the usual design a single language model makes both, by emitting a call or by declining to emit… 36 r/LocalLLaMA community 27d ago Qwen 3.8 27b is a amazing model, for the first time I see a local model found its own away to open a browser and test I was testing this quantization IQ3_XXS from GSQ-RCO with PI. It is a heavy quantization case, the model is in IQ3_XXS and KV cache in (Q4_0, Q4_0). I asked it to make the flight simulator, using that popular prompt. For my surprise, when I went verify the session I saw some… 35 r/LocalLLaMA community 28d ago "I'm the one degenerating" -qwen 3.8 27b q4km mtp I was working with my local llm on llama.cpp to get another server up with vllm, but we were running into trouble with the tool calls. When I asked it to investigate it went into a doom loop. Then during the troubleshooting of the doom loop it suddenly became self-aware. Another… 20 r/LocalLLaMA community 28d ago What's the current best LLM uncensoring method? With the recent Nvidia Huggingface acquisition and frontier AI labs screaming about safety and putting guardrails everywhere, I think it's important that we have local models that aren't affected by arbitrary guardrails set during training. To be clear, this post NOT about… 28 r/LocalLLaMA community 28d ago You can offload most of Qwen3.8-Flash-Next's KV cache to RAM with little decode slowdown I'm pretty sure it can be done with any model based on qwen4exp, which Qwen's next local models will be based on. You can use a quant that barely fits in VRAM and still run at the model's maximum context length without kv cache quantization, since most of the KV cache can live… 5 arXiv — Machine Learning research 28d ago Structural Negative Transfer in Federated Graph Neural Networks: Diagnosis, Causal Investigation, and the Limits of Divergence-Aware Mitigation arXiv:2609.16977v1 Announce Type: new Abstract: Federated learning lets multiple participants train a shared model without pooling raw data, by exchanging locally trained model updates instead. Federated averaging assumes that averaging local models is a reasonable way to solve… 4 r/LocalLLaMA community 28d ago Connected a local model (Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q4_K_M) to GIMP via MCP tools using llama.cpp - and here's the image result from my first prompt "can you draw a picture of a flower in gimp?". Needs work. Setup follows. can you draw a picture of a flower in gimp? So I ran through this setup to install the MCP server suite from GITHUB, it's pretty easy to follow the install - and installs the libraries necessary to connect llama.cpp to GIMP. https://github.com/maorcc/gimp-mcp After that's… 20 r/LocalLLaMA community 28d ago What's the next local model you are excited about? And why?   submitted by   /u/MrMrsPotts [link]   [comments] 35 arXiv — NLP / Computation & Language research 29d ago When Consistency Does Not Mean Reliability: Evaluating Local LLM Judges Against Human Ratings arXiv:2609.13824v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used to evaluate the responses of other language models. This approach, known as LLM-as-a-Judge, is faster and cheaper than human evaluation. However, a judge may produce consistent… 10 r/LocalLLaMA community 29d ago If you have a 3090, or other 30xx for local LLMs, I have something for you I have a custom fork of llama.cpp designed around the ampere architecture specifically (though many of the upgrades also translate to faster performance of blackwell + lovelace). The recommended config supports 90+ TPS (for agentic/coding, at temp 1; greedy will of course be… 10 Hugging Face Daily Papers research 1mo ago Building and Evaluating Fixed-Voice Thai TTS from Synthetic Speech Abstract A compact Thai text-to-speech model is distilled from a large voice-cloning teacher using synthetic data from brief voice references, achieving strong on-device accuracy and prosody with minimal errors. Generated by thinkingmachines/Inkling-Small In low-resource… 33 arXiv — Machine Learning research 1mo ago On-Device Language Models for Privacy-Preserving Stress Prediction: A Multimodal Evaluation on Mobile Health arXiv:2609.11961v1 Announce Type: new Abstract: Stress is a pervasive determinant of mental health and a key target for mobile health interventions. On-device language models (ODLMs) offer privacy-preserving inference without cloud dependency, yet their feasibility for health… 35 Page 2 of 10 · 500 articles ← Newer Older →
