It is that it makes pages load slower because browsers do not cache SSL content unless the Cache-Control HTTP header is set to Public.
When some of the content in a SSL page, like images, have that header set to Public the browser will show a warning stating that some of the content on that page is not secure frightening the user.
So what will a site owner do? 1) Use SSL without browser caching and make the site much slower; 2) Use SSL setting some content as cacheable and risking loosing its users or; 3) Send everything in the open making the ignorant user happier?