Unity definitely had a shaky launch. There is much hatred for it, whether it's warranted anymore or not.
Ubuntu's driver issues aren't purely a Linux kernel issue (although it might be in some cases), because they have their own installers to detect what drivers should be downloaded/installed.
The one thing I've always had issues with was sound. Not directly drivers, but in the past PulseAudio was a pain to keep working. This was mostly fine for about 5 years, before the 2010 editions broke things for me on the desktop again. I've had a much more pleasant desktop experience once things are installed properly in a VM. Using VM auto-installers always left me with really broken locale/keymap settings, and manual installations still give me a US keyboard layout during layout, which is bloody useless for the majority of the world.
It's all fixable, but sometimes searching for the solution will give you conflicting threads. It helps knowing how it all works in the background. I'd say Ubuntu was easier to install and use than Windows at some point (several years, in fact), but the quality of the installer has dropped lately. Internationalisation issues, sound and possibly older hardware are the weak points suddenly. Linux has historically been the best solution to squeeze new life out of old gear, so this is a sad development.
If you're installing a server version of any distro, you'll rarely have any issues at all. Of course, if you are doing that you hopefully have the knowledge to sort out the few issues that could crop up ;)
I'll typically start with just a bare server version of Ubuntu for a new computer, then apt-get the desktop environment+package manager frontends I like (both command line and GUI) once I have it recognising the sound hardware. But you shouldn't have to :/