Building an open-source control plane for self-hosted vLLM, what would you want in it?
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
Every time I self-host a model I rebuild the same stuff: start the container, set up a route, check why it died overnight, remember to shut the GPU off before it burns money.
So I'm building a panel that handles it. Start/stop models, OpenAI-compatible endpoint, health checks with auto-restart, basic usage and latency stats. No Kubernetes. Works with local GPUs or rented ones (RunPod, Vast.ai). MIT licensed and self-hostable.
Not released yet, so I'd rather ask now than build the wrong thing:
- What breaks for you when self-hosting inference?
- Do you swap models often, or pin one and leave it running?
- What would you need before letting it manage a GPU unattended?
Happy to hear "just use X instead" if this already exists.
[link] [comments]
More from r/LocalLLaMA
-
The AI race: China's open-weight models trail US closed models by just 7 months
Oct 14
-
Anthropic Executive Says Open-Source AI Models Are Higher Risk
Oct 14
-
ComfyUI v0.40.0 released
Oct 14
-
So this is how they do it --> RTX 5090, once a consumer gaming card, is now on smugglers’ menu alongside cigarettes
Oct 14
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.