'Optimise the common case' has been wisdom in the *nix world from before we were born. In the case of X, what was the common case when it was designed is not the common case now, and X does not optimise that common case.
That common case is local display, using graphics hardware which is built around 3D and OpenGL.
What the display subsystem needs to do is to efficiently make available the display hardware capabilities of the machine it is running on, in a way that is easy for people to program.
Then there is the question of what a modern desktop environment needs and how to efficiently deliver that.
The design assumptions of X, and the need to work around things using extensions and suchlike, make things harder than they need to be.
If you want your 'evidence', take a look at the size of code and execution time required to do basic and complex tasks using X vs similar situations on Mac and Windows. X solves problems that don't often need solving, and is a poor fit to the situation where it is most often used.