Comment Re:Communications overhead? (Score 1) 9
For tensor-level parallelism, you're only recombining the activations, not the weights, so the required bandwidth is far less.
I use TB RDMA to connect 2 MBPs currently for this purpose.
As for speed, the link is ~80Gbps (minus overhead). It's significantly better than realistic network connections, and critically, it's DMA- it's DRAM-to-DRAM transfers.