News / #edge Tag Edge 407 articles archived under #edge · RSS Sign in to follow r/LocalLLaMA community 1mo ago MIT LLM Serve Dashboard I am making open source A single-file, dependency-free live dashboard for your local LLM serving box — GPU utilization, per-model throughput, KV/context fill, and system stats for llama.cpp and vLLM , in one green terminal-styled page. No framework, no build step, no external requests. The frontend is… 31 r/LocalLLaMA community 1mo ago A game about Simulation Theory that includes an LLM. I made a game called "Simulation Simulator" . It's a freeform conversation game where you try to convince your AI best friend that reality is a simulation and you're inside a video game. The game has a local LLM packaged inside of it that you can run entirely offline. Been about… 37 r/LocalLLaMA community 1mo ago Should I get a 5090 or a strix halo? tl:dr; RTX5090 for 3400€ or Bosgame M5 AI for 2500€? I have a fairly new computer with a 5080 and 64Gb of RAM. I've been having loads of fun with local LLMs. In the end I find myself using thu free Claude and Deepseek V4 Pro through their API because it's so fucking cheap and… 9 r/LocalLLaMA community 1mo ago Has anyone created a "Local LLM Survival Kit"? Here's what I'm thinking about: A USB thumb drive that you can plug into any PC or laptop, and immediately get a usable knowledge base powered by an LLM, without requiring an Internet connection. I believe the technology for this should be ready. Rough architecture: llama.cpp… 21 r/LocalLLaMA community 1mo ago I built barebrowse: give a local-model agent a browser without Playwright — pruned ARIA snapshots instead of raw HTML (far fewer tokens) Author here, sharing something I built. If you run agents on a local model, feeding a whole page as raw HTML burns your context fast. barebrowse turns a URL into a pruned ARIA snapshot — the semantic tree with nav/ads/boilerplate stripped — so each page is a fraction of the… 10 Hugging Face Daily Papers research 1mo ago A Quantized Native Runtime for On-Device Semantic Audio Generation Abstract A dependency-free runtime enables efficient text-to-music generation on embedded devices through quantization and activation steering while maintaining audio quality. Generated by Qwen/Qwen2.5-Coder-32B-Instruct Semantic audio applications increasingly require… 22 r/LocalLLaMA community 1mo ago Neuralwatt Pricing will Double From 07/16 - Got This Email Welp, the age of cheap tokens coming to an end. Many here use this service to access GLM 5.2. Looks like only sustainable way moving forward to local LLM.   submitted by   /u/BoogerheadCult [link]   [comments] 26 arXiv — Machine Learning research 1mo ago Rethinking Small VLM Quantization: From Component-Wise Analysis to Hardware-Aware Edge Deployment arXiv:2607.08029v1 Announce Type: new Abstract: The emergence of vision language models with fewer than 3 billion parameters has accelerated the implementation of on-device multimodal intelligence. However, a detailed understanding of component-wise quantization remains a… 12 arXiv — NLP / Computation & Language research 1mo ago Different Teachers, Different Capabilities: Sub-1B On-Device Distillation for Structured Text Enrichment arXiv:2607.08268v1 Announce Type: cross Abstract: High-volume structured extraction pays a large model's latency on every item, so distilling the task into a small on-device model is attractive: comparable output at a fraction of the time and cost. We measure what that… 11 Hugging Face Daily Papers research 1mo ago CineMobile: On-Device Image-to-Video Diffusion for Cinematic Camera Motion Generation Abstract CineMobile enables efficient image-to-video generation on mobile devices through distillation-guided pruning, diffusion distillation, and hybrid quantization techniques while maintaining visual quality and achieving significant speedup. Generated by… 6 r/LocalLLaMA community 1mo ago If You Already Pay for an LLM Service, Running Local Embeddings and Rerankers Feels More Useful Than Running Local LLMs https://preview.redd.it/v0xtn3jdu9ch1.png?width=2047&format=png&auto=webp&s=628a6a541fe5f097d0f771ae0ba3b7f44126198f https://preview.redd.it/vjxiucsdu9ch1.png?width=2047&format=png&auto=webp&s=74f7a18a5a30276e206e2bfb5a0c529826ce86e4 This post was originally written in Korean,… 33 arXiv — Machine Learning research 1mo ago Robust Federated Learning Under Real-World Client Churn arXiv:2607.06979v1 Announce Type: new Abstract: Federated Learning (FL) enables training shared models on private, on-device data, but production deployments remain constrained to slow, multi-day refresh cycles due to the complexity of coordinating massive client populations.… 9 arXiv — NLP / Computation & Language research 1mo ago Billions of Sketches Reveal Hidden Cultural Variation in Human Concepts arXiv:2607.07267v1 Announce Type: cross Abstract: Claims about the universality of human concepts have been predominantly assessed through linguistic similarity across languages and cultures. However, words are effective as communication devices because they compress rich… 34 r/LocalLLaMA community 1mo ago Can you explain the concept behind each of the main size ranges of LLM models, as in, what hardware setups the different size niches are meant to fit into (~30b, ~70b, ~120b, ~230b, etc). Like is it mainly based on pro hardware sizing for 8-bit, or consumer GPU vram for ~Q4, or… I am curious about intended sizings of the main size niches of the popular local LLM models. As in, we can see there is a major niche at 26b-35b, then hardly anything from 36 through 69b, then (formerly) another major niche at ~70b-72b, then another niche at ~120b-123b, then… 22 r/LocalLLaMA community 1mo ago What GUI-first coding tool tool are you pairing your local LLMs with? Opencode isn't it for me. I've grown very frustrated with OpenCode. The web GUI and desktop app ideas are good, but the execution not so much. The GUI is lacking so many basic features. It's clear that the TUI is more important to the devs. Is there anything free that provides a more feature-rich GUI?… 14 r/LocalLLaMA community 1mo ago Complete local model asset generation pipeline So I figured I'd update the community given I just shipped a nice little feature set and feel like sharing it finally :) In the past few weeks, I've been test-coding an isometric RPG game/engine in Three.js, as part of my research into how LLMs work at scale in higher quality… 20 r/LocalLLaMA community 1mo ago The standard free ChatGPT LLM you get after a few messages HAS to be some sub-20b model with online search enabled, no other way to explain how awful it is Recently bought into the local LLM hype by buying a 32gb vram gpu and holy shit, gemma 4 31b at 5bits blows the standard ChatGPT model out of the fucking water. I just can't unsee the quality difference now that I've experienced it. Does Openai just cut costs by serving their… 34 r/LocalLLaMA community 1mo ago Can you trust local models to answer accurately? My goal is to improve as a developer, thus I needed to know if local llms can answer technical questions accurately The conclusion is that without rag they don't do too well, but with rag they are very good. Thinking didn't really help, and took so long I only got the scores for… 29 r/LocalLLaMA community 1mo ago I made a tool that chains a small local model into a big coding model and auto-unloads VRAM between them A couple weeks ago I shared PromptChain here a small Streamlit app that chains two models: a little Prompter that rewrites your rough idea into a proper prompt, then a larger Coder that turns that prompt into code. The whole point is that on an 8–16 GB card you can usually only… 18 r/LocalLLaMA community 1mo ago Are there any local ASR models that surpass Whisper right now? Hey everyone, I'm currently using faster-whisper(medium/large turbo) for local speech recognition, running it on an 8GB VRAM GPU. It works great, but I was wondering if there are any new open-source/local models that outright beat Whisper at this point? Here is exactly what I'm… 18 arXiv — Machine Learning research 1mo ago STELLA: Efficient Sensor-to-LLM Translation for On-Device Human Activity Recognition arXiv:2607.03089v1 Announce Type: new Abstract: HAR is increasingly expected to run continuously on edge devices, yet recent LLM-based methods remain hard to deploy: raw sensor prompts are long, cloud inference adds latency and privacy risk, and fine-tuned LLM pipelines turn… 33 arXiv — Machine Learning research 1mo ago Decentralised Federated Learning over Temporal Networks: The Role of Heterogeneities arXiv:2607.03171v1 Announce Type: new Abstract: Decentralised federated learning, based on peer-to-peer communication, is increasingly proposed for on-device training of machine learning models, promising a privacy-preserving, communication-efficient training process with no… 30 r/LocalLLaMA community 1mo ago UPDATE: I built a tool to turn your Claude Code sessions into fine-tuning data for local models (You can now convert your Codex and Pi sessions) A few days ago I shared this resource I created to convert your Claude Code sessions into training data (Thank you so much for all the support :D ): Original Post Today I'm sharing that I just released version 1.5.0, which now supports converting your Claude Code, Codex, and Pi… 13 r/LocalLLaMA community 1mo ago Running Gemini nano locally. You know that chrome lately downloads a local model (Gemini Nano). Which probably is a gemma vision quantized model. I tested it inside the browser, but I wonder, how could I load the "weights.bin" file from a linux shell? Both tensorflow and llama.cpp fail to recognize it. It… 10 r/LocalLLaMA community 1mo ago Local models + big context = slow. How are you orchestrating "map-reduce" style agent workflows? I tried running local models (qwen3.6*, ds4 flash, gemma4*, etc) on my mbp pro m5 with 128Gb of unified memory and concluded the bottleneck is context size. The moment a conversation gets long (16k is already the bottleneck), inference slows to a crawl. If you work with Hermes… 35 r/LocalLLaMA community 1mo ago Prefill vs. decoding and local LLM ROI: is prefill underrated? I'm trying to understand why, when people discuss the ROI of running LLMs locally, they almost always focus on output speed (decoding) and rarely on input speed (prefill), which seems like it could have a significant impact on hardware ROI. Yesterday I saw a post on X where… 7 r/LocalLLaMA community 1mo ago OpenComputer | An Open Source Computer Built For Agents. Open Computer running in an isolated VM with inference running M4 Pro via LM Studio Gemma 4 13B QAT Hey everyone, Tim from AnythingLLM , where we have been building productive an on-device agent and AI assistant experience for the past 2.5 years now. I want to talk about a new… 15 r/MachineLearning community 1mo ago Edge AI ASL Recognition on Raspberry Pi 5 – Looking for Feedback on My System Design [P] Body:   submitted by   /u/Unlikely_Let_9147 [link]   [comments] 6 r/LocalLLaMA community 1mo ago <15B local models to test on medical benchmark? HI, in a few weeks the medical exam for italian residents will drop, and I would like to compare small models that run on my laptop against it. It would not be a standard benchmark for llms but it would give a sense on how much better does it perform compared to the average… 7 r/LocalLLaMA community 1mo ago Is DeepSeek v4 (Flash) really extremely cheap to run? If yes, how? Hi. I don't have a GPU. So my biggest "local LLM" experience has been running ~26B models with single-digits tps values. However, the "serving economy" of DSv4 models look like a riddle to me. The Flash model has 284B parameters, but providers (e.g. OpenRouter) charge so little… 20 r/LocalLLaMA community 1mo ago Local LLM for legal-document adaptation keeps hallucinating citations with total confidence — grounding/model/pipeline ideas? ## Why local is non-negotiable I'm a criminal defense lawyer and the documents I'd feed this contain **client-confidential case material**. Sending real case content to a cloud API isn't an option for me, so this has to run on my own hardware. Frontier models are clearly better… 13 r/LocalLLaMA community 1mo ago What do you guys use local models for? I previously built a telegram bot that chatted with people. But it was never put in production. Besides that I often find that even the cheap tier premium models like ChatGPT 4.1mini are a little bit too stupid to do things. They just don't seem to grasp the job as well as the… 27 r/LocalLLaMA community 1mo ago Who Has The “Jankiest” Local LLM Setup? | Non-Official | Fun Contest | No Prizes Had an idea for a fun no prize/non official competition to see who has the “Jankiest” local LLM setup. NOTE: This is NOT an official competition. There are NO prizes. This is just for fun . Rules: 1. One Submission via comment per person 2. Has to be your current setup or your… 23 r/LocalLLaMA community 1mo ago I built an opensource AI notepad alternative to Granola, need help to improve it Hey, I built a project called steno. Steno is an AI notepad for confidential conversations. It runs fully locally on your device with local llms like Gemma 4 quantised. The quality has gotten pretty good now so wanted to share to the communities like LocalLLM that helped me… 26 r/LocalLLaMA community 1mo ago Built Kivarro, an all-in-one local inference workbench. Looking for brutal feedback from people who actually run models locally. I’ve been building Kivarro , a desktop app for local model inference. The idea is simple: Most local LLM tools solve one slice of the workflow. One app for chat. One app for model files. One script for llama.cpp flags. One dashboard for memory. One terminal for logs. One random… 36 r/LocalLLaMA community 1mo ago I built Kivarro, an open-source Rust/Tauri local LLM inference workbench. Looking for r/LocalLLaMA testers. Hey r/LocalLLaMA , I’m the creator of Kivarro. It’s an open-source local model inference workbench built with Rust/Tauri, focused on running GGUF models locally with llama.cpp, profile switching, model registry, runtime controls, local API status, logs, benchmarks, and a… 22 r/LocalLLaMA community 1mo ago A planetary test for local models Here is a fun test prompt: Imagine a date in the next 1000 years where the Sun, along with its gravity, suddenly disappeared. When that happens, all planets in our solar system would stop orbiting and carry in a straight line. Is there a date that would then cause two planets… 34 r/LocalLLaMA community 1mo ago I benchmarked 13 models at 65K-128K context to find out what actually matters for agentic workloads I benchmarked 13 models at 65K-128K context to find out what actually matters for agentic workloads — prefill dominates everything, and KV head count beats parameter count I've been running local LLMs for agentic workflows (tool use, coding agents, RAG) and kept seeing people… 36 r/LocalLLaMA community 1mo ago Ran a classic(medival europe) fantasy RP/agentic benchmark across 8 local models Qwen3.6-27B held up better than its size suggests Threw together a benchmark suite (quest completion, scene endings, item/time tracking, character detection, storytelling, drafting) and ran it across 8 models people talk about a lot on here. Judged with an external LLM grader, N varies per category (shown on the chart). Overall… 32 r/LocalLLaMA community 1mo ago Using local models with Hermes vs Claude code Today I saw this in StepFun’s blog for their Step 3.7 Flash model. Running the model with CC performed better results vs Hermes. Curious why?   submitted by   /u/GreatMammad [link]   [comments] 34 r/LocalLLaMA community 1mo ago A fully local, self-hosted repo index for coding agents (Rust, MIT, runs offline) If you run local models for coding, feeding them repo context without blowing the context window is half the battle. I built basemind to index a repo locally and serve it over MCP: a code map across 300+ languages, git history and blame, and document RAG over 90+ formats, all on… 15 r/LocalLLaMA community 1mo ago Qwen3.6-27b-mtp-q8 successfully created an A* pathfinding implementation on a test game built in Java from scratch. I used Claude Code locally running this model. Took a surprisingly long amount of time to pull off. This game is just a test game my friend is making and we're both vibecoding. He uses Codex due to lack of hardware for local models to do the graphics, UI and lighting, while I… 9 r/MachineLearning community 1mo ago Small Language Model SLM [D] Hi, I am supposed to prepare for SLM and its software part for an on campus internship, i've worked with local models like ollama generally,in my projects and also with open claw so can anyone guide me the last 2-3 days tips on what should i go through for this internship prep??… 19 Hugging Face Daily Papers research 1mo ago DuoMem: Towards Capable On-Device Memory Agents via Dual-Space Distillation Abstract DuoMem is a dual-space distillation framework that transfers procedural problem-solving from large language models to compact student models through context-space and parameter-space distillation, achieving high performance with minimal additional parameters and… 31 r/LocalLLaMA community 1mo ago Mapping Local Nodes - Mildlyinteresting I've been working on mapping (with tags) and steering local models based on their activation path in specific context to questioning during a/b testing. There is no insight or "how to" here, no benchmarks or improvement suggestions, no products. I just think that the activation… 23 r/LocalLLaMA community 1mo ago Follow-up: DeepSeek V4 Flash on 2x RTX PRO 6000 finishes real coding tasks faster than Sonnet and Opus, at about Sonnet quality This is a follow-up to post about which local models stay fast deep into long context and I learned a lot from people here. I kept measuring after that and it turned into a proper indie coding bench. With DeepSeek V4 Flash running on vLLM it lands around Sonnet quality and it… 24 r/LocalLLaMA community 1mo ago Claude Code and China: The mechanism is activated when the user sets the ANTHROPIC_BASE_URL environment variable (used for local models) The list of suspicious hostnames is not stored in plaintext within the code; instead, it is Base64-encoded and then encrypted using a simple XOR operation with a key of 91. Once decoded, it reveals domains belonging to Chinese companies, keywords related to artificial… 38 r/LocalLLaMA community 1mo ago I made a local model that gives you a multimedia role playing experience (and you can too!) https://huggingface.co/lemonade-sdk/RPG-HaloTales-V1   submitted by   /u/jfowers_amd [link]   [comments] 17 r/LocalLLaMA community 1mo ago I built a local LLM NPC backend focused on NPC-to-NPC conversations I just released a research project I did last year as open source. It is a fully local speech-to-speech backend for LLM NPCs. So speech-to-text, local LLM, text-to-speech, no cloud needed. The main focus was NPCs talking to each other, not just answering the player, and my study… 17 r/LocalLLaMA community 1mo ago Palantir CEO rages against closed models For context, this week they struck a deal to buy Nvidia chips and run local models for their enterprise clients. So in this video he is railing against Anthropic and OpenAI saying they are ripping everyone off while stealing their data too. Always a special moment when the enemy… 30 Page 4 of 9 · 407 articles ← Newer Older →
