Maybe it's the applications. As you note, we have proved, time and time again, that in contrived scenarios Java code can meet or even beat the performance of so-called "performant" languages like C, C++, Lisp, VB6, or JavaScript. And when presented with such evidence, most sane, open-minded opponents will hang their heads in shame and promise never again to spread such slander.
...but then, they fire up Eclipse, or NetBeans, or Guiffy, or enable the Java support in their browser, or try to run an app on their favorite feature phone. And they wait for it to become responsive...
...and wait...
...and wait...
...and wait...
...and wait...
...and...
...what did I promise never to do again? Sorry, must have dozed off...