Inside Amazon's Data Centers

1sockchuck writes "Amazon Web Services usually doesn't say much about the data centers powering its cloud computing platform. But last week the company held a technology open house to discuss the company's infrastructure, sharing cost data and a glimpse of a modular data center design. The key point: AWS is growing like crazy. 'Every day Amazon Web Services adds enough new capacity to support all of Amazon.com's global infrastructure through the company's first 5 years, when it was a $2.76 billion annual revenue enterprise,' said AWS Engineer James Hamilton, whose presentation (PDF) is available online."

Re:

[fuzzyfuzzyfungus]

I think that it was manager-safe enough. There were a few diagrams that you should be careful to skip over(a few had percentage numbers that weren't finance related!); but the rest had numbers with dollar signs in front of them safely ensconced next to a variety of recognizable corporate logos with a generic-but-high-tech photo of some server racks in the background.

A few good buzzphrases as well. I'm definitely going to ask my department about why they aren't being more proactive and action-oriented abo

PDF is heavily slashdotted

[Anonymous Coward]

Please use Coral Cache [nyud.net] to reduce the load on the original.

Re:

[jewelises]

Someone should host the PDF on S3.

Re:

[Anonymous Coward]

They should have hosted it in the cloud.

Re:PDF is heavily slashdotted

[jewelises]

Coral Cache isn't working for me, but google has cached [googleusercontent.com] it.

Re:

[l_bratch]

I have mirrored the PDF here too: http://www.tghost.co.uk/JamesHamilton_AmazonOpenHouse20110607.pdf [tghost.co.uk]

What good is that

[Anonymous Coward]

"Every day Amazon Web Services adds enough new capacity to support all of Amazon.com's global infrastructure through the company's first 5 years, when it was a $2.76 billion annual revenue enterprise,"

What good is that when their network design is so flawed that a single human error can bring the whole service down for a week?
Amazon Web Services? No, thanks, I prefer reliability.

Re:

[0123456]

Do you have a link to that incident?

He does, but unfortunatley the cloud server is down.

Re:What good is that

[nereid666]

http://tech.slashdot.org/story/11/04/29/1156220/Amazon-EC2-Failure-Post-Mortem [slashdot.org]

Nope

[aclarke]

The anonymous coward claimed that "the whole service [was] down for a week". The link you posted isn't even remotely close to that.

any reason they don't buy larger servers?

[alen]

presentation said average is $1450 per server which is an entry level 1U. why not just buy fewer larger servers to share the workload? it would probably save on power costs?

Re:

[PickyH3D]

I'd bet that buying the cheaper, yet still powerful computers is probably more cost effective considering the rate that these machines burn out under their kinds of load. Why spend so much money on a machine that will be broken, when you can just buy multiple cheap ones instead? Then, when one burns out, you're still running at a higher capacity even with the negative of higher power draw.

Re:

[tibit]

They didn't say if it's per a real or virtual server methinks, and I don't know if the assumption that they pay whatever anyone else would be paying holds water either. I'm sure they can get a lot more "server" for $1.5k than me or you would.

Re:any reason they don't buy larger servers?

[micron]

Looking at a 3-5 years TCO, and power costs where these data centers are located, power costs are noise in the equation.
Taking advantage of commodity pricing in the lower tiers is where the savings is at. Example, single socket systems are a lot cheaper on the procs and mainboards than dual sockets. Quad socket processors are significantly more expensive per proc..
At $0.10 per KwH, a 400W server is $350/year to power. Quad socket processors (Intel I7) can be as high as $4500 each!

Re:

[SuperQ]

Don't forget to factor in PUE into the power costs. An average datacenter has a 1.5x multiplier for power. Over 3 years a 400W machine will cost about $1500 to power.

your calculation is flawed

[dutchwhizzman]

You need to factor in the cost of cooling those KwHs too. Not just that, but there are large area's of the planet where you can't buy a KwH for 0.10 US dollar. Try 3 times as much for an average European country. You'd be looking at two to six times your $350/year easily for running your 400W server just in power costs. Also, those $4500 CPUs won't cost that much for large companies like Amazon and Google. Rest assured they pay probably substantially less than $1000 for those CPUs. These companies are big e

Re:

[micron]

Your points are exactly why these large data centers are locating in areas with access to cheap power. I used $0.10 as an example, however, that is extremely HIGH when factoring in the deals that large data centers strike with regional power providers that are giving cheap access to hydro power. This is the exact reason folks are not putting large data centers in Europe and the Bay Area. Power has to be cheap for the economics to work out.
Also, read the papers published by Google and Facebook. These guys ar

Re:

[micron]

They pay the per 1,000's price (or close to it), which is not list price. These numbers are published. This is the quantity in which they purchase the processors, so it makes sense. The number you can't see is the negotiated power deals.
The price for a quad processor capable system is still 4x to 10x what a single socket processor costs.

Real world at scale.

[Anonymous Coward]

Larger servers are prohibitively more expensive than el cheapo low-end servers, but fail at effectively the same rate.

As for power costs, no, not necessarily - unless your large consolidation boxes are underpowered. Most of the 4U boxes I've dealt with have massive power requirements.

Re:

[mlts]

I would disagree at the same rate figure.

What you pay for with the bigger servers is redundancy. The higher end servers that Oracle and IBM offer cost more, but they engineer for reliability, not absolute cheapness of price as in the commodity x86 market. Yes, you can improve uptime by adding redundancy on upper layers up to and including the backend app.

On one end, you have FB's solution where reliability isn't as much as issue as deploying fast. The top layer backend app handles the redundancy. On the

FPGA

[Lennie]

The problem with networking is, you need dedicated hardware (ASICs) to able to push the packets at those speeds.

Those ASIC's are not the same for every vendor, I wouldn't be surprised if their are big differences in implementations and very strongly tied to the 'control plan' (the software running on the management-CPU).

Ofcourse the control plane could be a lot more open.

The closest things I can think of are the open source software which can be used to program FCGA's: the NetFPGA and liberouter project do

Am I the only one?

[kannibul]

Am I the only one that thinks that cloud computing is really NOT going to take off? Amazon is spending a boatload of money, in the hopes that other people are going to be willing to offload a boatload of money...in the hopes that they can process data faster/better/etc off-site. Just the security issues alone concern me, but the additional bandwidth is what really gets things going... Just saying - from a business perspective (and my perspective as a network admin), it seems that Cloud Computing is a 90's

Re:

[donnyspi]

Yeah, the thing is it already has taken off. They can hardly add additional capacity fast enough. AWS is a very profitable business for Amazon. Our company switched from a traditional data center setup to running fully on AWS and it did reduce our costs and increase our agility and scalability. That said, like anything it's not a fit for everybody.

Re:

[kannibul]

I agree that it would have to be something very adaptable to usage with cloud-based approached...still, even where I work (Oklahoma) we're on a tornado-prone area, and I work for a small "State and Local" government agency. We considered moving our datacenter to a location that is rated for EF5, flood proof, redundant this and that, and would safehouse our data entirely from any imaginable threat except nuclear. The costs were reasonable as well for what we'd co-locate...but what killed it was the need for

Re:

[silky1]

I would think staffing a team to maintain hardware in an IT Department is much more expensive than paying Amazon to run the backend. Thinking of many corporations, all have data centers, IT staff, hardware, software etc, and here is Amazon with a single data center capable of providing all this infrastructure at what has to be a competitive price for them to survive. This has to be a better option for many companies, especially small companies. My behemoth company is constantly looking for off shoring op

Re:

[noahm]

Well, they did say that "Every day Amazon Web Services adds enough new capacity to support all of Amazon.com's global infrastructure through the company's first 5 years". They wouldn't be doing that if it wasn't taking off.

Keep in mind that "cloud computing" is a really vague term that is used to describe all sorts of different technologies and platforms. It's still really immature, in terms of best practices and that sort of thing. There are still a lot of people throwing around the "cloud will solve all o

Re:

[mlts]

It hasn't just taken off, it has already made a sonic boom that has rattled eardrums of anyone in virtually every IT shop worldwide.

Every company out there hears that the mystical Cloud can solve all computing problems, has no security issues, etc. Reality hits when the realization comes into play that choosing a cloud provider means a permanent relationship -- it is virtually impossible to change providers due to each having different APIs. The fact that one will have to pay for a data center somewhere a

Re:

[alen]

it's not a reduction in costs as in a reduction in up front cash required. in finance class in college we learned that a fast growing company can easily go bankrupt due to the fact that cash comes in months after the revenue is recognized but expenses like salary and bills have to be paid out on time.

in this case small companies can pay a little at at a time to amazon and grow rather than spend a fortune on infrastructure

Re:

[dohnut]

As people have said, it already has taken off. Personally, I use AWS for much humbler reasons than most. I use it for hosting my personal domain(s) -- web, DNS, SVN, etc. I used to just have a linux server in my basement do it all, but then you have a $500 machine to maintain, a static IP to provision, possible TOS violations with your ISP, poor upload speeds, etc. For a few dollars a month I can host everything I used to at home on a virtual linux server with redundant storage attached and excellent ba

Re:

[NateTech]

Mainframes never worked. ;-)

It's just another swing back to what was originally promised out of the 90s "client/server computing" buzzword.

"Thin clients", and keep the data and processing power on the centralized servers.

The "new" part about this "cloud" stuff is that instead of needing to ask for your own "timeslices" on a big shared computer, they created a way to stuff your own virtualized OS on the thing, and made the "big server" out of a bunch of little ones.

Nothing really all that new here. The main

What is amazon doing different?

[Synerg1y]

I've seen a few different data centers, they all look about the same, why should we care? I'm pretty sure it was SUN Microsystems who came up with the modular shipping container design, which was back in the day.

It's over 9000!

[synapse7]

Is this the same web service systems that couldn't support the demand(how much could it be) for lady GaGa's album?

Somebody missed ...

[Anomalyst]

the opportunity to crow about how effective it might be in generating bitcoins.

The only problem I have with Amazon

[MJMullinII]

Is that they are, IMHO, extremely hypocritical in how they conduct themselves.

On the one hand, they put themselves forward as this oh so "wholesome" family environment, while doing absolutely NOTHING to stop their "Pro" sellers from listing the most vulgar items you would care to search for.

These pages are also listed with the helpful "Sell yours here!" but if you actually set up an account and attempt to sell said items yourself, you run the definite risk of having your account terminated by what might as