True, it's not about network access per se.
Whether you are using "ssh -X" or just using DISPLAY to point your application to another machine, it is useful because of two properties in the X protocol, which Wayland does have.
Property 1: Although we routinely use shared memory extensions in modern X setups, a lot (including the core functions to which all applications must be able to fall back) works over a socket, which can be a unix local socket or a TCP socket.
Property 2: The X11 protocol has a slew of very sophisticated features which enabled graphical applications to work around the latency of the communication and to reduce communication bandwidth. An application can store gylphs in the Xserver and then, referencing those gylphs, it can draw nice anti-aliased text using a very small amount of bandwidth.
Wayland lacks both properties, but property 2 is the big issue.
Redirecting X11 applications over a network doesn't work well if the application sends a ton of commands synchronously, unless the network is low latency (eg, local LAN).
Redirecting X11 applications over the network doesn't work well if the application sends the entire graphics window as a single pixmap, untless it's a high bandwith network (eg, local GbE LAN).
Wayland works over a Unix socket too. And you can set which socket the application will attach to with WAYLAND_DISPLAY.
The first problem is that the protocol assumes shared memory but it would have not complicated things that much to make it work without requiring shared memory.
But the real problem is that the only method Wayland applications have to draw is by sending the entire window (surface) as a single pixmap. Works wonderfully over shared memory, works like crap over my cell phone's 3G connection.
A 800x600 surface rendered at 60 fps requires 86.4 MB/s.
Adding to Wayland the rich set of server-side rendering features would complicate things too much for Wayland to be possible.
And it would be a somewhat futile exercise because, increasingly, X11 applications are doing less server side rendering and more pushing of large pixmaps.