The problem with that is most browsers don't respect cache headers. The spec is a little vague on what's supposed to happen with HTTPS... some say "cache nothing!" but that's convention and not due to any restriction from the spec. If the stuff you're sending out is static (images, css files, external javascript, etc.--i.e., the stuff that takes the longest amount of time to download) the site should be able to set the expiration cache to a year and expect the browsers to respect that.
That simple action alone would speed up https traffic to a level where people would fine it completely tolerable, and in most cases virtually no different than regular http. The problem is, most browsers see HTTPS and they ignore all the explicit headers and you're back to requesting that same stupid 150K background JPG image over. and over. and over. and over... it's fucking infuriating.
Seriously, browser makers: fix this shit and we can ditch unencrypted HTTP. The most insulting thing is that Internet Explorer is the only browser that gets this right.