The posts suggesting you don't re-invent the wheel are wise, especially since there are already packages out there to get you where you want to go with negligible overhead. Check out xubuntu for a pretty much out-of-the-box barebones GUI without spending a week or two (or more) learning the minutiae of Linux inner workings, or if you really want to sink your teeth in, get Gentoo running and only include packages for xfce and mono. Gentoo is pretty much a build-your-own-distro kit (at least last I used it over a year ago).

The only thing I found Gentoo could do reasonably better than Windows or a pre-built distro or even GNOME/KDE on Gentoo, ever, in months of messing with it, is that I could run a Playstation emulator (ePSXe) with no "hiccups" due to background processes kicking in every now and then. Even then I likely just am not configuring something right or I could have solved the problem with a newer and speedier processor. Believe you me, I was looking for applications of "barebones" setups--I had a huge nerd-on for only running the bare minimum of what I needed and thought it would be this vastly superior experience. It's really not.

Really, the minimalism thing is only for learning how things work, very specific situations with limited hardware power, and embedded systems. For a media center with most hardware made in the last like 5 years (or longer, depending), xubuntu will be more than enough minimalism.