Optimal settings for 2 GPUs in LM Studio
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
Hello everybody. I've got a 5070ti and a 5060ti both 16GB in my system which is a 5900X and 64GB DDR4 RAM. I'm trying to run some 16-18GB models like Qwen, Cydonia, Skyfall.
I'm having problems utilising the VRAM I have to get the best usage out of it. LM Studio sees the 32GB VRAM but regardless of if I use Tensor parallelism, Split evenly or Priority order I always get an error after waiting for about 5 minutes for the model to load.
The pattern is always the same: The loading progress bar for the model starts pff quickly then crawls in the last 5-10%. Then I get an error reporting that the model couldn't load.
Does anybody have any tips for optimal settings to get the best out of my system? I know that having 2 GPUs doesn't magically mean you have double the memory and there's caveats. But nevertheless I've also read that LM Studio does have the capability to leverage those 2 GPUs to improve speed.
(EDIT) I should add that I've been trying context lengths of 16384, 32768 which LM Studio is saying will use 17 GB of VRAM so well within the reported 32GB I have. I've even had it working occasionally but most of the time the model fails to load.
Many thanks for any help.
[link] [comments]
More from r/LocalLLaMA
-
The AI race: China's open-weight models trail US closed models by just 7 months
Oct 14
-
Anthropic Executive Says Open-Source AI Models Are Higher Risk
Oct 14
-
ComfyUI v0.40.0 released
Oct 14
-
So this is how they do it --> RTX 5090, once a consumer gaming card, is now on smugglers’ menu alongside cigarettes
Oct 14
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.