News / #edge Tag Edge 500 articles archived under #edge · RSS Sign in to follow llama.cpp releases dev-tools 1mo ago b10950 ggml-cuda: fallback to F32 on device without BF16 hardware acceleration ( #28846 ) ggml-cuda: fallback to F32 on device without BF16 hardware acceleration: (Nvidia >= AMPERE, AMD >= RDNA3 or = CDNA) apply logic to NVIDIA as well Co-authored-by: Johannes Gäßler johannesg@5d6.de… 23 r/LocalLLaMA community 1mo ago best local model for Japanese translation right now? I've been thinking of translating some light novels. Have 24GB VRAM.   submitted by   /u/RadianceTower [link]   [comments] 11 r/LocalLLaMA community 1mo ago The Local LLM community feels like the golden era of the internet all over again Lately because of the current hardware shortage, unfortunately or fortunately, we can’t just throw infinite cloud compute at our problems, but we’re forced to actually care about what’s happening under the hood. We’re tweaking inference engines, learning quantization math, and… 32 r/LocalLLaMA community 1mo ago What pi.dev plugin do you suggest for context, compaction and memory management of local models? I have been battling with my Qwen3.8:27b setup on my rtx 5080 16gb. I am using llama.cpp to run a nvfp4 version of qwen3.8:27b llama-b10699-bin-win-cuda-13.3-x64\llama-server.exe -hf williamliao/Qwen3.8-27B-NVFP4-GGUF:NVFP4 --jinja --chat-template-file… 13 r/LocalLLaMA community 1mo ago I fine-tuned a 2B LLM on our WhatsApp group chat, and shared how to do it on GitHub as a cookbook. https://github.com/Sayitobar/chat_llm_cookbook This is my personal project that took several months. I wanted to see whether a 2B small local model could simulate a six-person group chat trained & ran on an M1 Pro. How good is it?: - It's fun, but not great. It doesn't achieve… 6 r/LocalLLaMA community 1mo ago This is why we need open-source harnesses + local models i've been thinking about this more after trying different agent setups. the model isn't the only thing that determines how well an agent performs. The harness around the model matters a lot too. With a managed agent setup, you're often giving up control over things like the… 21 MIT News — AI research 1mo ago Lifesaving Lincoln Laboratory device wins 2026 Excellence in Technology Transfer Award The handheld catheterization device AI-GUIDE, created by Lincoln Laboratory and Massachusetts General Hospital, promises improved health outcomes for injured service members and civilians. 9 r/LocalLLaMA community 1mo ago 7900 XTX + 32/64GB RAM for Qwen 3.8 Flash Next? Planning to build a PC mainly for local LLMs/coding agents. I keep seeing 3090 + Qwen 3.8 Flash Next benchmarks, but could not find enough info for the 7900 XTX 24GB . 3090s are hard to find where I live, while newer Nvidia GPUs are too expensive, so I am considering a 7900 XTX… 38 arXiv — NLP / Computation & Language research 1mo ago X-CoSD: Communication-Efficient Cross-Vocabulary Collaborative Speculative Decoding arXiv:2609.09166v1 Announce Type: new Abstract: This paper investigates collaborative speculative decoding (CoSD), a distributed large language model (LLM) inference framework in which an on-device small language model (SLM) drafts candidate tokens and a server LLM verifies… 30 arXiv — NLP / Computation & Language research 1mo ago From Fixed Keys to Readable Schemas: Small Language Models for Vehicle Agent Function Calls arXiv:2609.09476v1 Announce Type: cross Abstract: In-vehicle assistants must translate natural-language requests into accurate vehicle function calls under strict memory and latency constraints, making small language models (SLMs) attractive for on-device deployment. For such… 27 arXiv — NLP / Computation & Language research 1mo ago PELM: Power Efficient On-Device LLM Inference with Speculative Decoding and Dynamic Voltage Frequency Scaling arXiv:2609.09662v1 Announce Type: cross Abstract: Deploying Large Language Models (LLMs) directly on mobile platforms at the edge is gaining traction due to a myriad of benefits, such as increased privacy, personalization, and reduced latency. However, LLMs have heavy… 27 r/LocalLLaMA community 1mo ago Local LLM / Qwen 3.8 win I’ve been working on perfecting my setup ever since 3.8 hit, but no longer pay for subscriptions. I had a coding interview today and setup openrouter ahead of time as another option. I figured I could use the latest GLM flash for cheap and it would be fast. Nope. Over thought… 9 r/LocalLLaMA community 1mo ago Would you consider 5t/s usable for a local model? I'm able to run qwen3.8 27b in two ways on my system: split between my 3060 12gb and 9070xt running at 20t/s or running off the 780m iGPU and 5400mhz DDR5 at 5t/s. Personally I feel like the 5t/s is still more usable because I have enough RAM to still use my system mostly… 4 r/LocalLLaMA community 1mo ago Don't let FOMO win if you're interested in local llm from a hobby/learning aspect Just a reminder for those out there itching to get into local llms - don't let FOMO or "gear acquisition syndrom" take over. No matter the hobby, it's so easy to get stuck in a trap where we buy more trying to do more only to realize we've lost the fun in it all or even the… 21 TechCrunch — AI news-outlet 1mo ago Apple CEO John Ternus says the best AI device is still the iPhone The company also argued that its on-device models offer consumers more privacy. 8 r/LocalLLaMA community 1mo ago Desert Ant Labs: On-device intelligence for every product   submitted by   /u/Arcuru [link]   [comments] 38 Hacker News — AI on Front Page community 1mo ago Desert Ant Labs: local, fast models that run on device Article URL: https://desertant.com/blog/introducing-desert-ant-labs/ Comments URL: https://news.ycombinator.com/item?id=49624823 Points: 205 # Comments: 61 31 r/LocalLLaMA community 1mo ago Qwen3-0.6B (400 MB) on a Samsung Note 8 (2017) phone drives a real desktop Chrome Up front: I'm one of the people building the page-perception layer used here. We started by testing small local models. The result turned out to be more interesting than the original test. 12 small models, 3 verifiable tasks, logs, and offline replay. Setup: Galaxy Note 8 (2017,… 17 r/LocalLLaMA community 1mo ago Which local model is actually good at knowing when to stop and ask you a question? I’ve been thinking about this after using more agentic/local coding models. A lot of the newer models are surprisingly good at continuing on their own. But sometimes that seems like the problem. If a requirement is ambiguous, I’d rather the model stop and ask: “Do you mean A or… 18 r/LocalLLaMA community 1mo ago I made Warrior Quest, a local LLM-powered dark-fantasy RPG where the model only plays NPCs and the actual game state stays deterministic The LLM is limited to NPC emulation. Game state, world logic, quests, and the authored story are handled by deterministic game systems rather than the LLM. I built it this way because I wanted the freedom of talking to NPCs like you would at a tabletop game, without handing the… 7 r/LocalLLaMA community 1mo ago After over a year of my nights and weekends, the Jenny app is done! Hi all! I just wanna say that I am tired lol. Yes, it's another harness, but I spent a lot of time and effort and have forsaken my hobbies to build the Jenny (like XJ-9) app. Jenny is a free, MIT licensed electron desktop app for running local LLMs with tool calling, rollback,… 14 r/LocalLLaMA community 1mo ago 9 easy steps for llama.cpp, a local model, Freecad (and pi coding agent) to generate solid objects that sound mechanically good and can be also be 3D printed/milled Quick setup on linux: STEP 0: install/download llama.cpp, Freecad, your favourite gguf model - possibly with multimedia image reading capabilities (I've used Qwen3.8-27B-UD-Q4_K_M and relative mmproj-F16 quantized by Unsloth), uv (, pi.dev) STEP 1: $ cd /your/path/to/ (i.e.… 6 r/LocalLLaMA community 1mo ago Easy local Copilot with VS Code and Lemonade Not so long ago I wrote a guide on how to get GitHub Copilot running with a local model in Visual Studio Code. Since then Copilot subscriptions have got much more expensive, local models have got much more powerful and getting local Copilot up and running has got much easier. So… 6 r/LocalLLaMA community 1mo ago Best local models for hardware programming? Guys can you tell me which local LLMs are best for hardware programming? like Verilog RTL, UVM, System Verilog?   submitted by   /u/Curious_Cantaloupe65 [link]   [comments] 11 r/LocalLLaMA community 1mo ago What is the obstacle in front of Local Frontiers? We've reached a point with local LLMs where models are now very close to (and even reach) the level of models like the Opus, with some minor modifications. While some K3 and GLM 5.3 models are incredible, they are barely as powerful as the Opus or on par with the Fable or Astra.… 7 arXiv — Machine Learning research 1mo ago PACE: Propagation-Aware Collaborative Correction for One-Shot Personalized Federated Graph Learning arXiv:2609.04832v1 Announce Type: new Abstract: Client heterogeneity creates both an opportunity and a risk in personalized federated graph learning. Knowledge held by other subgraphs may complement a receiver's Local model, but an incompatible transfer can override reliable… 18 r/LocalLLaMA community 1mo ago Trying to create my own server and consuming it for code with my phone remotely (Mac OS) Hi there! I need some help with this. I have a 32gb Macbook Pro with the latest available update of Tahoe. I'm using LMStudio with MLX to serve a local model and I want to expose it so that I can consume it with my phone to code and review stuff when I'm commuting to places.… 28 r/LocalLLaMA community 1mo ago 2x R9700, 64 GB DDR5 is an absolute beast machine with vLLM Radiance / R9V and Qwen 3.8 27b and Flash next I've been tinkering with local LLMs since the beginning of the year when I had an Intel Arc B580 and 32 GB of DDR5. Curiosity got the best of me and I bought the first R9700 about half a year ago, also because I wanted to upgrade my gaming graphics for 4k. As the 5090 was about… 14 r/LocalLLaMA community 1mo ago Planning to get a cheap-ish GPU. Would appreciate some advice. Hi, I've been wanting to run my own local LLM for some time and I finally saved enough to get a budget GPU. I can spend 700$ at most, and I'm looking for a GPU that can run quantized 30B~ parameter models with decent speed. Being able to run Qwen 3.8 27 B Q4_K_M and similar… 14 r/LocalLLaMA community 1mo ago Validate your local LLM advertised KV cache against real pressure; see exactly how old contexts get evicted from cache Hello, I'm a bit obsessed with cache management on local LLMs. For the last few days I've been working on cache management on vLLM with my 2x DGX Spark and DeepSeek v4 Flash 0731. I felt something was off so I investigated, found issues, fixed them, but needed a way to validate… 22 r/LocalLLaMA community 1mo ago How does your favourite local model do with the slinky test? Prompt: Make me a single HTML file of a rainbow slinky going down an up-escalator forever. No libraries, just canvas and code. The slinky should be a chain of springs, each coil a different color of the rainbow. It starts folded in half like a horseshoe draped over a step. When… 27 r/LocalLLaMA community 1mo ago My local LLM demoscene generator can now watch its own output and rewrite it! I've updated my auto_demo_scener project with Ninfer support and a “rewrite based on video” feature that I thought you might find interesting. The project is basically an endless demoscene machine. A local LLM writes Three.js effects (from a library of editable prompts), you… 23 r/LocalLLaMA community 1mo ago Any resource on using Blender with local models, and which models work best? Hey all, I've seen some really fun looking things with people having their local models drive Blender to create pretty cool looking world scenes. Is there a good tutorial on setting up Blender yo be driven by your model? For example, what programming harness, do you use a MCP… 22 r/LocalLLaMA community 1mo ago Is there a local LLM or toolchain to edit 3d models? I got a lot of ads for meshy recently and went to try it with hilariously bad results. It apparently can't do anything but decorative figurines. I wanted a body shell for an rc car and it just couldn't generate a car without wheels or bottom chassis. It also looks like it can't… 22 r/LocalLLaMA community 1mo ago I've found myself using Local LLM's like 3D printers. Anyone who has a 3D printer and get use of it finds it incredibly useful for those odd jobs around the house, a missing bracket, a cable router, steam deck holder and so on. In the past if I was missing an app or useful software, a game I'd do the lazy thing, even though I can… 22 r/LocalLLaMA community 1mo ago Qwen3.8-27b is the first Local model im able to blindly trust You know that thing where you just throw a task at a frontier model and not have to supervise it worrying of it going off course? Qwen3.8-27b has officially gotten me to that point for local work. He has been doing non-stop continuous agentic work for 8+ hours and hasnt screwed… 14 arXiv — Machine Learning research 1mo ago LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference arXiv:2609.03079v1 Announce Type: new Abstract: On-device LLM inference is attractive for privacy and responsiveness, but remains challenging on mobile and embedded devices because model weights far exceed available DRAM. Prior systems exploit activation sparsity and offload… 36 r/LocalLLaMA community 1mo ago Bernie Sanders proposes to ban AI Defined as AI exceeding human cognitive abilities. 20 years in prison. Plenty of local models already fall under that big of an umbrella in some capacities. This is why it's not enough to say that you could torrent open models so who cares what the politicians do. They want you… 4 r/LocalLLaMA community 1mo ago I built a local web UI to finetune models on my own text and actually watch the training (works on AMD ROCm) I wanted to do continued pretraining/finetuning of a local model on my own notes and see what's happening while it trains and do it on my AMD card, since most tools assume CUDA. llm-training-panel is a local web UI that: - loads a model from a local dir or a HF id - shows a live… 34 r/LocalLLaMA community 1mo ago Can a 4B local model actually feel like an AI assistant? I've been building Arcon around Qwen3-4B + LoRA. Instead of just making it a chatbot, I'm experimenting with persistent memory, personality/mood, internal state, tools, and eventually having it process things before replying. I'm curious what people who've built local agents… 6 r/LocalLLaMA community 1mo ago KV cache might be a bigger problem for local models than parameter count Everyone keeps talking about fitting larger models into local hardware, but parameter count isn’t the whole story. For long context inference, KV cache can become the real memory bottleneck. Every new token adds key and value states that need to stay available, so a model that… 8 arXiv — NLP / Computation & Language research 1mo ago How Do Prompt Variations Affect Energy Consumption in On-Device LLMs? arXiv:2609.01798v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed on mobile devices, making energy efficiency a key deployment constraint, yet the energy impact of prompt design remains underexplored. This paper aims to understand how two… 4 r/LocalLLaMA community 1mo ago Chrome browser add-on that uses local LLMs to move thousands of unsorted bookmarks into a smart list of automatically calculated categories Is anyone interested in my Chrome browser add-on that uses local LLMs to move thousands of unsorted bookmarks into a smart list of automatically calculated categories?   submitted by   /u/paranoidray [link]   [comments] 7 r/LocalLLaMA community 1mo ago What do you do in the meantime when your favourite local model is thinking and working hard with your harness? Real question, whether your are vibe coder or expert or whatever, what do you do? Read carefully each single word? Plan the next steps? Do the gym?   submitted by   /u/takoulseum [link]   [comments] 14 r/LocalLLaMA community 1mo ago GLM 5.3 Flash makes a black hole Minecraft mod running locally on 4x RTX PRO 6000 WS saw the post the other day where people said Minecraft clones aren't impressive anymore, because at this point the whole thing might as well be in the training data. so i tried something slightly different, which is asking a local model to write a mod for the real game, using… 5 r/LocalLLaMA community 1mo ago Qwen 3.8 Flash Next for Creative Writing? As we all know on of the best local models for creative writing is gemma 4 31b and Muse Glimmer 30b. However, ive been a happy user of Qwen3.8 Flash Next and I wanted to know how well Qwen 3.8 Flash next is doing in terms of creative writing (preferably German).   submitted… 19 Hacker News — AI on Front Page community 1mo ago My local model setup on an M4 Pro Mac Mini Article URL: https://lws.io/blog/my-local-model-setup/ Comments URL: https://news.ycombinator.com/item?id=49529132 Points: 244 # Comments: 151 36 r/LocalLLaMA community 1mo ago All currently popular local models in one table + Opus 4.8 results If you are thinking what model will fit best your HW specs and tasks you are doing here is one table with all currently popular models that still can be considered as local. LLM Test Scores Feature DeepSeek-V4-Flash-Vision-Exp DeepSeek-V4-Flash-0731 Qwen3.8-Flash-Next… 24 r/LocalLLaMA community 1mo ago Which current local models that can run within 128GB generate the best SVG pelicans? I used a famous Simon Willison's pelican riding a bicycle prompt on the biggest local LLMs that can run on 128GB Apple Silicon. U used quantizations by Unsloth. Qwen3.8 Flash-Next gives a lot of details. DeepSeek V4 Flash is strangely underwhelming. Qwen3.8 27B still rocks, and… 6 r/LocalLLaMA community 1mo ago First time running local models Sad that I only have 12gb of vram but this ik_llama is so fast   submitted by   /u/Needausernameplzz [link]   [comments] 36 Page 3 of 10 · 500 articles ← Newer Older →
