The layer in the system between the user applications and the hardware interface is the place where QT, GTK, Windows graphics api, and all the other graphics toolkits go. Those toolkits shouldn't care too much about the hardware details, just the published capabilities of the GPUs.
It's kinda hard not to care, because a lot of it depends on where you have the data, where the processing capability is and what the link capability is. Sending a video stream is heavy. Even sending an event stream like applications do all by themselves is too heavy during say a resize or scrolling action. Some time ago I experimented with turning Qt into a remote application toolkit, basically taking all signals and slots and serializing them over SSL. It was actually surprisingly successful, basically it was puppeteering a client to draw the interface and using signals and slots to synchronize information on demand. Only the bits you connected sent events across the link.
There were plenty little gotcha's though, like the scrolling I found a way to make a trigger that'd only fire after a custom delay, like for example 50ms after you were done resizing. And I needed to add a system to say "When this button is called, include the check state of this radio button and text of that textbox", but the nice thing was that on the client side it was acting like a client window. It was resizing, the menus were popping up, the buttons responded (though the actions might take time due to latency) and I could do client-client signal/slots like "when the user checks this box, enable these extra fields" too without a server round-trip.
I could do neat things like send a jpg and have the client draw it and even if the client moved the window around, covered it with other dialogs, scrolled it in and out of sight it was zero overhead. Yeah I know kind of like a browser, but not like any RDP/VNC solution. Often I needed the same resources over and over and didn't want to transfer them every time, so I needed a caching system. Kind of like HTML5 persistent objects I guess but before that. And I could populate list/tree/grid objects up front or on demand, a bit like DOM manipulation in HTML.
It wasn't transparent but it was somewhat API transparent, you'd get a "RemotePushButton" instead of a "QPushButton" which acted the same, but instead of actually drawing anything just sent commands to the client which drew the real QPushButton. You didn't really see that though, you just called the functions and connected the RemotePushButton's onClick() signal as if it were a QPushButton. Kind of like HTML+AJAX on steroids but looking and feeling like a native application. That I feel would have been rather next-gen to see it finished.
If you're wondering why I didn't then mainly because the product I was thinking to use it for kinda died on the drawing board. And because to really become user friendly it'd have to integrate on a much deeper level, so you could use all the Q* classes without rewriting everything, the Remote* classes were a hack (QObjects) working with the standard library. And you'd want to put more work in persistence, the idea was that you could yank out the plug on one machine, log back in on another and it'd redraw everything but initially it passed all though and didn't shadow the client state on the server. It could have though, the rewrite was just too much.