my knowledge might go a little deeper than yours because I had an interest in computer graphics many years ago. Although I've not got a perfect understanding, I'm probably more right than wrong (or somebody here will tell me otherwise and we can both learn a thing or two).
but ultimately, the way that people write high performance graphics stacks now favours compositing and graphics cards require to do a lot of direct accesses in order to make that happen as fast as possible, you have a region of memory and it's mapped to a location somewhere in the gpu memory and you just blast it with data, you can't have too many processes in the middle here, so thats why we need a DRM (direct rendering manager) to basically give x a direct path to the graphics hardware without having to go through the cpu or the kernel etc, as little as possible anyway.
Anytime you have to context switch, you lose time, which hits performance hard, so everything is like cleared out the way and the software and hardware almost talk directly to each other, which doesn't really happen with any other software, writing files, accessing network, computing data, almost always goes through the cpu and the kernel and for graphics applications which render megabytes of data per second, this is just awful and dramatically kills the performance. The retina display on an ipad is 2560 x 1600, so at 24/32 bit colour, each complete frame is 11-15MB, imagine 60 of those? Easily 600 - 900 MB per second. All of that data is being transferred from the system to the gpu every second the display is on, so it's a huge amount of data. Of course, you can reduce this by using gpu memory buffers and "damages" to know which surfaces need to be re-rendered (i.e. rendered and transferred again cause the display information inside them changed)
However, this kind of "direct, get out of my way, let me speak to the manager" way of thinking doesn't really fit with the linux way of doing things, everything is integrated, the x protocol isn't really to blame, but the only reason for people wanting to keep x alive is because of the protocol, it is basically a way to draw pixels, lines, primatives, AND their favourite, abstract everything across a network so software can be anywhere and X will solve how to display it. EXCEPT IT DOESNT SOLVE IT!! Even when you want to network x, you have to manually setup a bunch of configuration on both machines to get it to work, it doesn't work by magic, or autoconfiguration, cause nobody cares enough to do it, evidently, cause even now you have to manually do this, 20 years later.....what a bullshit system....
Of course, then you get to the lower levels and a whole bunch of compromises come into effect, nobody agreed on even the most basic things, fonts, oh lets make a font server, which will never work properly because of all the compromises made for that and it'll only work in certain circumstances....
It really is the most bullshit, fucking stupid system ever created and people seem to be so stubbornly ignorant and stupid they refuse to kill it, cause I dunno, it's really hard to understand, you see some of the hatred for wayland from these people are you are just so flabbergasted by it, you can't even start to reason, why would you keep x alive? it's almost like it's their child, they wouldn't care if it had three arms, ate other babies and urinated acid and used it as a weapon, they STILL wouldn't kill it....or even contain it...cause it's their baby....Thats the kind of logic you're going up against if you talk to these idiots...
But it's such an ugly, nasty, knarly mess of bullshit and compromises that NO DESKTOP UI TOOLKIT USES IT, Qt, Gtk and Wx for example, pretty much just request a drawing rectangle and then do everything themselves, completely sidestepping x, but if you looked at the x protocol and especially a "unix haters" entry on the subject and know something about computers and/or programming, you'd understand why people want to kill it so badly.
Take a look at this and then try evaluate that against the people who want to keep it alive...Of course, some of the explanations here maybe fixed, or resolved in some small way, the text is very old, but I hope it gives you a great insight into why people hate X so much....Take it with a pinch of salt and humour, I hope it works for you like it did for me.
http://www.art.net/~hopkins/Don/unix-haters/x-windows/disaster.html
WHY THE FUCK WOULD YOU WANT THIS PIECE OF SHIT TO SURVIVE?? KILL IT WITH FIRE AND REPLACE THE WHOLE SHITBAG
Thanks for letting me rant, I hope in some way it was educational, even though the language perhaps was not :D