Yeah I second this caveat –you DO have to install everything from the ground up, but frankly it's worth the time to learn to install a LAMP or Rails stack, and doubly worth the time to optimize away all the stuff you don't need. Besides, Apache administration is fun!

I just got a Linode account and I like the service a lot. I also use Slicehost for several small sites, which is almost identical in terms of service offernings (although a little more expensive bang-for-the-buck-wise), but reliable, and you get snapshot backups for $5 extra/month.