I'll apologise in advance for rambling but I don't often get to talk about Plan 9 and it's nice to have the opportunity!
X11 itself as files would, I imagine, become a bit icky because it's a complicated protocol. But as I understand it, the Plan 9 windowing system was effectively exposed as files (i.e. the display server exported a filesystem interface to applications) and that did actually permit some pretty cool stuff...
Windows basically appeared as a set of files that let you draw to the surface within the window. The interface exposed to windowed apps by the windowing system could also be consumed by other instances of the windowing system, so that nesting instances of the windowing system into windows Just Works.
The fact that everything was in the filesystem meant that network FS shares could be used to e.g. to provide rootless graphical remoting of applications, etc - you just had to arrange for the right filesystem mounts to be available and the display would automagically work out.
Having said all that, obviously these days you'd also want to worry about direct hardware access to GPUs, etc, which I'm sure would make the whole enterprise rather more complicated! Maybe that would put paid to the idea of "everything is a file" being practical for display stuff, or maybe somebody cleverer than me could point out a better way of doing it!
Further tangent: Plan 9 made device files really behave like files, which meant you could remote device access trivially using remote filesystem mounts also. This doesn't work with Unix device files and it always seems a shame that various ad-hoc remoting protocols (USB over IP, Network Block Devices, etc) get used instead. But I suspect a similar argument to the GPU could easily apply - that it's either more efficient to have a specialist protocol or that some devices are just too complicated to meaningfully abstract like that. Who knows.
In some ways, I think I "miss" the olden days when if you'd got a Plan 9 system you probably could feel justified in believing you were in the future!