Ok... In reverse order of significance:
1. Expense (at least traditionally) of SSL Certificates. Although today that's not such a big issue, and an SSL certificate costs about the same as your domain registration. However, if you have multiple subdomains you still need a more expensive certificate.
2. Complication. It can be a highly confusing process for someone who's not an expert to do the certificate request process and the associated installation of the certificate. I know the first time I tried to do it, it went horribly wrong and I spent a day trying to sort it out.
3. Client Performance: SSL websites are slower than non-SSL websites. Not such a big deal again these days, but I remember when I had to wait it would seem forever for the images on my online banking site to load, cursing them every time for such a graphically-intensive SSL site.
4. IP addresses: This is a big one, if you host multiple websites on your server, and you only have a single IP address, you can't host more than one SSL certificate. So you need more IP addresses (which is not easy nowdays). Big deal for small company hosted websites, which are often on shared IPs.
5. Server Performance: A server that can cope with one million users per day using HTTP will not be able to cope with anywhere near that number of connections over HTTPS for obvious reasons. So you not only need a certificate, you potentially need a whole new server architecture to deal with it. Scale this up for a big business like Twitter or Facebook and you're talking implementation costs in high millions of dollars.