The transport mechanism between a local X client and the X server is not restricted to a Unix socket. The X server vendor is free to implement it as a shared memory queue or such (I believe Accelerated X servers do something of that sort), so this is not a flaw in X architecture.
Your idea about mmapping the framebuffer and allowing Xlib to deal with it directly is completely flawed, OTOH. First off, it disables use of hardware acceleration, which means performance would drop much more than using a Unix socket for communication with the X server. However, even if you have some sort of plugins for the Xlib which provice acceleration for specific cards, there would have to be a single semaphore for serializing access to the card's accelerator between different X clients.
People who designed X were not stupid and really, really experienced. X does have flaws, but not where you showed them.
Anyway, all of the above applies to "classical", 2D operation mode of X. Direct 3D rendering for games is something absolutely different, and your ideas, transformed to some point, are correct :))
Heck, Precision Insight had them half a year ago :))
Cheers,
-diskena