First you should ask yourself why there is a whole separate distribution just to support a different desktop.
Because Xubuntu users want to have only the Xfce desktop, without having to install the Unity desktop first, which, if you're never going to use it, means you're just wasting hard disk space.
We have had a recent lesson in this fallacy in the case of Mint. Mint is also a copy of Ubuntu and it exists primarily as a platform for the Cinnamon desktop. But because they were slow to handle security problems, Mint was hacked and code compromised. I don't trust Mint to this day. So I suggest starting with a secure and solid Ubuntu base and just perfect your desktop on that distro.
What an odd point of view. Linux Mint got hacked through Wordpress running on its web site. They weren't "slow to handle security problems"; they dealt with it as soon as they found out about it, which was almost immediately. And If you had checked the MD5 checksum of the hacked ISO, you would have seen that there was a problem with it.
As its leader, Clement Lefebvre, wrote in response to a comment on his blog, "...we’ll probably also contract a security firm to look into the bottom of this for us, we’re software developers not intrusion experts."
Take your idea to its logical extreme, and we would just have one Linux distro with a number of different desktop environments. Nobody wants that, except you, maybe.