I agree that we need to come up with a brand new system to handle today's graphics systems. That's what Wayland is for, and why it's such an interesting project. It is not legacy baggage, but a ground up designed system. You have heard of it, haven't you? Seems like every Linux user and their dog knows about it these days.
Also, I'm very glad that Wayland is implementing an X compatibility layer. I'm one of those fraction of a percent that use and enjoy network transparency. It would annoy the hell out of me if I had to run a full graphic system on the servers I manage, and then use VNC to connect to them. It's just so much nicer to ssh into the machine, run the program, and have it appear directly on my screen. Never mind that I like keeping a minimal amount of packages installed on the servers. I try to keep it simple.
By the way, if we have the memory, computational power, and bandwidth, why are you so worried about X overhead and latency? Surely they become marginal with more resources.