I certainly agree the desktop is the most important part, at least for adoption and market share. The problem is that Unity is a terrible UX and much worse than what it replaces. I thought pre-Unity Ubuntu was pretty good; not great but competitive. In fact I have a Ubuntu desktop that all my family shares as an alternate to Windows, pre-Unity even my 10 year daughter was fine using it. But when they switched to Unity it became much harder to use, especially for folks who were trying to use it as an alternative to Windows, Mac and prior Linux desktops.
I saw a different thread recently (lost the URL, sorry) pointing out that Microsoft is making a similar UI design error on Windows 8, i.e. they are trying to repurpose a mature mouse-focused UX to become good at tablet touch interface also. As we have all heard Win8 is also is unsuccessful. So it is a shame that Ubuntu has abandoned focusing on a high quality mouse-driven UX, because if that was available during Win8 launch for first time desktop Linux would be well positioned to pick up market share from Windows.
I have developed apps for lots of UIs over the years (web, PC, Mac, various X variants, and lots of mobile including Palm OS, iOS and Android). You can't just re-skin widgets to make a UI work for both touch and mouse. They are functionally different UX that require a deeper UI abstraction via MVC or similar.