llama-launcher v1.3 release -> Bayesian Optimisation
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
| Hello everyone, some of you may have seen a post of mine from a few days ago about my app, llama-launcher, a lightweight point-and-click GUI to create llama-server commands without the constant need for typing them up. Well, I've just added an optimisation feature that uses Tree-Structured Parzen estimation through optuna's framework. It uses llama-server to tune a pre-determined set of parameters to try to squeeze the last bit of juice out of your system, completely hands-free. I've been using this to get the last bit of performance from my MTP models without having to sit at my desk tuning, loading, prompting, and unloading manually and repeatedly. So far, I've seen upto a 15% improvement in speeds (as seen in the images) versus baseline commands with no tuning with Gemma 12B MTP during testing. Without any human interaction at all during the optimisation process. It's still in it's early stages so there are many improvements to be made but any suggestions you may have please let me know. You can check the repo out here: https://github.com/SolaryKryptic/llama-launcher [link] [comments] |
More from r/LocalLLaMA
-
Unpopular opinion Qwen 3.8 is hard to understand
Aug 30
-
Qwen3.8-Flash-Next turns 4xR9700 into a local AI powerhouse! 120 t/s TG and 12k t/s PP single request with optimized vLLM
Aug 30
-
Oh so that's where my PCIe lanes went...
Aug 30
-
Uncensored Multi-Model Releases, LongCat-Flash-Lite-Sparse with MTPs and LSAs, Qwen3.8-27B with MTPs, Qwen3.5-122B-A10B with MTPs, Qwen3-Coder-Next and Laguna-S2.1 with Vision, All Available in GGUF Format! Bonus: Links to my llama.cpp Fork for LongCat-Flash-Lite Support and…
Aug 30
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.