I came across this interesting article https://blog.exolabs.net/nvidia-dgx-spark/ while I don't have the DGX spark but it made me curious will this kind of arch speed up my setup for LLMs?

Mac can host large models but the prefill speed sucks, so I tested in it on my setup for Kimi 2.7.

Short answer: it helps prefill, but it does not meaningfully help decode on this setup. RPC is still mostly a capacity tool unless the network/interconnect and split mode are much better.

Setup

• Host: Mac Studio M3 Ultra, 512GB unified memory, Metal
• Worker: Linux box with NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 96GB VRAM, CUDA
• Network: direct Ethernet between Mac and Linux box, but only 1GbE in practice
• Measured RPC transfer rate: about 112-113 MiB/s
• Model: unsloth/Kimi-K2.7-Code-GGUF, UD-Q3_K_XL
• Model size on disk: about 432GB across 11 GGUF shards
• Runtime: llama.cpp server version 9827 (4c6e0ff3a), Unsloth build

Controlled test

Same synthetic prompt for both runs:

• Prompt tokens: 7120
• Generated tokens: 64
• temperature: 0
• ignore_eos: true
• Prompt cache disabled
• Prefill gain: about 14.8%
• Decode gain: about 4.2%
• Total request time improvement: about 12.3%

Split trend

The generation columns are - where I only ran prefill. The controlled generation rows used the exact same 7120-token synthetic prompt; the earlier split-sweep rows were around 7.1K prompt tokens but not always the exact same prompt.

20,80 was the practical max on this card with 128K context.

21,79 failed even at 8K context:

RPC/network trace

For the 7120-token prefill-only 20,80 run:

• Mac -> RTX: 251.59 MiB, 2.03s
• RTX -> Mac: 194.69 MiB, 1.49s
• Total RPC traffic: 446.28 MiB, 3.52s
• RTX graph compute: 1.34s

The RPC traffic is mostly hidden activations, not text tokens. For prefill it is chunked/batched, so the network cost is noticeable but not fatal. For decode, the boundary is crossed every generated token, which is why I expected decode to suffer more. In this test decode was roughly the same as Mac-only: 18.28 tok/s vs 17.55 tok/s.

Learnings

• I can knock off few more seconds by using a better cable, but not sure it's worth it
• It is useful for fitting models/splits that otherwise do not fit one device.

Question: As I was increase the shards, the prefill speed was decreasing, but will this trend continue if I add one more GPU? People with multi GPU setup what's you take on this?