I haven't implemented everything, but as long as a client doesn't send any references to objects the compositor doesn't have, it should work fine. I don't know how realistic this is, or how well it can be made to handle custom additions to the protocol. My contributions will be separate from wayland, more like wayland-demos, but it should be usable with other compositors; I don't know how varied the other compositors will be.
Wayland is just the protocol defined in a xml file. There is also a wire format specified in the docs, and there is no framebuffer information in there, that is done through shared memory. I am pretty sure wayland will always need something like RFB to mimic this shared framebuffer, in addition to the wayland socket, and another connection for drag and drop data.
Once you understand the internals of wayland, I don't think it would be a big step to make this remote client, so it is entirely feasible that Qt/Gtk could come up with their own.