Ah, cool. Apparently, they do have a VMWare product, and that's the one that I've ended up using. I hated it so much. If they have an alternative - Awesome. I know EC2 uses a very customized version of Xen.

Seems like VMware to me: http://www.rackspace.com/information/newsroom/pressreleases/rackspace-private-cloud-leverages-vmware-to-extend-enterprise-computing-on-demand/

RS = Rackspace? If so, they use VMWare, because it's ENTERPRISE!

What's wrong with AoE? It's high performance, low-overhead, and simple. We're looking at using KVM, or Xen. Xen PV has much better performance compared to our current ESXi cluster. I think our ESXi cluster is on FC + 1GE + HP Blades. We can get far better performance from Xen by using DAS and replicating it up to a AoE store in case of local drive failure. Drive latencies are 1/10th of what they are on FC, and we're able to get much better IOPs from using local SSDs. Our systems, and TCO is far lower cost by using commodity Dell hardware instead. Last company I was at had something like 500 TB aggregate storage, and 18 racks. We have over 100 Gbit/sec of aggregate bandwidth on our cluster. I'd say probably 25 gbit/sec of that was storage traffic. I'd like to see you try to meet that scale with a traditional FC SAN.

Don't they get their IPs from ARIN? They have to pay their ARIN dues like everyone else, don't they? Their endowment is 8.3 BILLION dollars. I don't think $160k is that bad for them.

I live in Scotland, and I recently signed up with an ISP that did this. They have block control over my building, so there wasn't any other solution. They have a device (I'm pretty sure it's just a linux box based on my nmapping) that looks at each packet's TTL. If the TTL is odd and the port is NOT 80 or 443, it drops the packet. If the TTL if odd and the port is 80 or 443, then it redirects you to a billing page. I bypassed it by incrementing TTL at my gateway. I imagine people will modify openwrt/dd-wrt to do this as well. Additionally, I have a solution which tunnels my connection over a VPN to an Amazon EC2 instance and does some magic to beat QoS. It seems like oversubscribing at at least 2000% seems typical. I'm paying for a 50 mbit/sec connection. I see closer to ~6-8 mbit/sec (no, I'm not getting my megabytes and bits confused). Additionally, since I know a little about my ISP internally, they run all these blocks to their HQ over MetroE-like products, and then concentrate it into their core. Their transit is 10GigE (based on traffic numbers pulled from various private sources). I know that they sell far more than 10GigE of bandwidth to customers. There is far more wrong with the ISP environment here than you'd know. My girlfriend's ISP is even worse. They do a significant amount of throttling on specific ports (1935, 80, so on..) and basically any kind of download during the day slows down to a halt. They have about 6000 people + several offices on a 10 GigE connection. I don't know if this will even be seen, but I thought it was worth throwing out there.