As someone who has multiple Nexus devices in his possession and has upgraded more than one of them to Lollipop, I can say that I rolled back because of UX problems. Some of the "optimizations" made in Lollipop are contrary to what you'd expect. Additionally, some of the new security features break existing functionality. I'm all for security (I tried four different ROMs on a device I got yesterday until I found one with working device encryption) but one should be able to relax security if they so choose.
Go test drive Lollipop in a VM. Some people love it, some don't. I won't be upgrading anything I own back to Lollipop until there are ways to work around some of the new features.
* Two finger/double scroll for notifications bar
* Complicated prioritized volume setttings (I believe they removed a volume override intent for this, which I use)
* Tabs and recent apps part of the same "recent apps" overlay (can be disabled via a chrome setting, thankfully)
* Removing long press on app bar buttons (changed to a small hot area of text under the icon, which I didn't know about until someone told me)
* Volume and brightness sliders in the notification bar
* Material Design
* Change to battery metrics display
* Heads up notifications instead of stealing the entire display for things like the phone UI (CM11 does this too)
As you can see, there's really no compelling reason to upgrade and some really annoying new features (as well as some rather critical open bugs) that will keep _me_ away from Lollipop.