Comment Re:Power? (Score 4, Interesting) 41
For now, the answer is to spread the datacenters across the country. Which causes other issues.
https://twitter.com/corbtt/status/1772392525174620355:
Spoke to a Microsoft engineer on the GPT-6 training cluster project. He kvetched about the pain they're having provisioning infiniband-class links between GPUs in different regions.
Me: "why not just colocate the cluster in one region?"
Him: "Oh yeah we tried that first. We can't put more than 100K H100s in a single state without bringing down the power grid."