SSL has two parts that take a lot of time - key exchange using public-key technology, which just depends on the number of connections, and data encryption, which takes time proportional to the amount of data encrypted. Until the last few years, the key exchange time dominated, because public-key operations are slow and most use of SSL was for encrypting passwords, credit card numbers, or other very small chunks of data. It was pulling teeth to get a lot of sites to use SSL at all (though the whole Certificate Authority system is a lot to blame for that), and it was pulling teeth to get a lot of sites to encrypt more than just your login and credit card data (such as the whole page that asks for your login.)
Do you think speed doesn't matter any more, now that lots of sites are running with the CPU relatively idle? How many SSL connections do you use where the server has bothered to turn on PFS, the Perfect Forward Secrecy stuff that does a one-time Diffie-Hellman exchange? (Appallingly few.) How many sites do you connect to that are using 2048-bit public-key or longer? (Some, but hardly most.) It's still about performance.