My intuition says that the network linking the two machines would be a bottleneck, but I don't know if that's a minor performance problem or a major one.
RAM bandwidth is the bottleneck, so one machine would actually be slower.
Apple Machine Learning Research published the following decode speeds for the NVFP4 checkpoint at an 8,192-token context (generating 128 tokens):
- Two Mac Studios, pipeline parallelism: 15.74 tokens/s
- Two Mac Studios, tensor parallelism: 24.99 tokens/s
- Four Mac Studios, tensor parallelism: 36.73 tokens/s
Mac unified memory is still slow compared to a NVidia GPU, but the MoE design makes it possible to run a large model.
The 1TB mac cluster is the price of a Tesla, while the NVidia rack will set you back the price of a Ferrari.