I have no idea what you mean by "poorly optimized drivers". The only things that an X server is expected to do and do well today as far as screen drivers go is to composite pixmaps generated by the painting backends in Chromium, Qt, GTK, etc. An X driver is not meant to do any drawing anymore - yes, the X servers still leave the old code paths around so that some obsolete app might use the X server to actually draw other primitives on screen. Nothing of note uses an X server that way anymore.
Given this, there's really no sense to X. Wayland with a VNC backend is all you need for remote work.