I'm running the Lars Op Den Kamp PVR build of Eden with tvheadend for the TV tuners on Oneiric and use a logitech rumblepad 2 for the controller. Set it up with zsnes and dolphin-emu and can access these without even leaving the couch. The addons repos are great, adding quite a lot of features you could only dream of in other media centres. I rarely have issues even with the latest un-stable builds and when I do there is already a patch out for it.
As for Boxee being for dummies, XBMC isn't hard at all anymore thanks to projects like OpenELEC. I built a system for a friend using OpenELEC and it could not be easier.