3 big issues with Unity as I see it:
Unity was pushed out before it was ready, just like pulseaudio, KDE 4 or Slashdat Beta was. People object to have their personal desktop turned into a dev testing lab. Even when/if the bugs are worked out, the bad first experience colors perceptions permanently.
Unity changed lots of aspects of how Ubuntu worked for no particular reason other than an eye towards touch screen usage. This is the same issue people have with Metro.
Unity's introduction also coincided with the "Amazon lens" feature that sent all local searches to Amazon's servers. This sort of invasive monetization is anathema to Linux users and seem like a blatant money grab by Canonical.