Agentic harness for small models
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
Hello; I'm a semi-beginner at local AI. I've been experimenting with this tech for a while, and I still haven't found a proper harness that fits my models, hardware, and needs.
My use case is pretty simple: web search, fetching, and browser use. Summarizing websites and having the agent explain stuff to me / Q&A. Potentially some agentic RAG (or whatever RAG-style implementation works best for my specs). Filesystem capabilities and a shell. MCP servers are a must (I use the GitHub, Hugging Face, Reddit, OpenRouter, etc. ones). A sandboxed JS/Python environment would be cool too. PDF reading is highly valuable to me, so that too.
So far, I've tried the following:
- A large LM Studio plugins/MCP roster. It works, but it's not ideal.
- Hermes Agent. It injects a ridiculous amount of context and does a bunch of prompt processing and other sci-fi stuff that slows the workflow down way too much.
My specs:
RTX 5050 8 GB VRAM
16 GB DDR5 RAM
13th-gen i5 CPU
My models (best I've found):
- Ornith 1 9B; Unsloth Q4 quant; Q4 KV cache; 65,536 ctx
- Gemma 4 E4B QAT; Unsloth Q4 quant; Q4 KV cache; 131,072 ctx
- Gemma 4 E2B QAT; Unsloth Q4 quant; Q8 KV cache or full precision; 131,072 ctx
- Ling 3.0 Tiny; using their custom llama.cpp fork; Q8 KV cache; Q4 quant; 131,072 ctx
- InternScience_Agents-A1-4B; Bartowski Q6 quant; Q8 KV cache; 131,072 ctx
All running with optimal sampling/settings in LM Studio.
Do you guys know of a lightweight harness that fills these needs? I want to finally settle this nonstop tinkering I'm doing so I can actually stick with something good and only switch models when better ones come out.
[link] [comments]
More from r/LocalLLaMA
-
Unpopular opinion Qwen 3.8 is hard to understand
Aug 30
-
Qwen3.8-Flash-Next turns 4xR9700 into a local AI powerhouse! 120 t/s TG and 12k t/s PP single request with optimized vLLM
Aug 30
-
Oh so that's where my PCIe lanes went...
Aug 30
-
Uncensored Multi-Model Releases, LongCat-Flash-Lite-Sparse with MTPs and LSAs, Qwen3.8-27B with MTPs, Qwen3.5-122B-A10B with MTPs, Qwen3-Coder-Next and Laguna-S2.1 with Vision, All Available in GGUF Format! Bonus: Links to my llama.cpp Fork for LongCat-Flash-Lite Support and…
Aug 30
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.