As someone who uses Ubuntu as their primary desktop OS both at home and at work, I have to say that usability is the biggest feature holding back Linux desktop.
I keep wondering about this one. Because of work requirements, I started using windows again after a long hiatus, and find it rather cranky (windows 7). It was easier to program the reactions to my marble ball mouse under linux than it was under windows 7 (essentially impossible to get reasonable scroll-wheel emulation). Then there isn't anything remotely comparable with xmodmap. I can't have multiple desktops. Files are named in weird ways (PROGRA~1, etc) that have their special rules (it really is much simpler in linux). The keyboard layout kept unhelpfully switching to whatever it felt was right, and it took a long battle to ensure it stays where I want it. And Skype has annoying ads under windows.
Installing updates is gargantuan pain in the buttocks, especially when compared with ubuntu. In windows, a reboot is almost always necessary after downloading and installing updates. Quite often you need multiple reboots, and all of it takes ages. Under ubuntu they are much faster and unintrusive.
So, in my experience Windows actually sucks compared to a decent linux distro. All the talk about the little annoying things in linux is, I think, due to an illusion. Windows is popular today because it was popular yesterday, so people are used to it and all its little (and not so little) annoying things. They just don't notice anymore.