AWS is hugely profitable, and most companies save money using AWS instead of running their own computers.
The reason really has to do with scale. Everyone knows that scale matters for the physical aspects (power and machines and buildings), but it matters for people possibly even more so.
If you're willing to build and staff a data center in the middle of nowhere it can probably be close to as efficient as Amazon's data center's - by "close" I mean well under 2x, not 5% or something. This is based on mghpcc.org, built by the 5 big Massachusetts universities - 15MW, $100M for the building, competitive power costs, room for 20-30K machines. Based on the public numbers, amortized over 20 years, the building costs less than $200 per 1U per year. At $0.10/kWhr (public tariff for MW rate * 1.2 PUE) power costs about $1/W/yr; our servers seem to draw in the 300W range or less. Since servers seem to last 3-5 years, that means the machines themselves cost considerably more than the building or the power, but not a huge amount more.
The MGHPCC costs about $5M/yr in depreciation; you could probably make the building smaller and cheaper, but the cost per computer would undoubtably go up, and there's not a huge amount of room before it starts adding a lot of per-computer cost - my wild-ass guess is that below half this price it would start getting really expensive. (I'd really need an industrial-scale A/C expert to make that prediction)
The other place where scale comes into account is your workforce, because people are expensive and they come in integer units. If you want to use serverless, and elastic mapreduce, and block and s3-like storage, and virtual machines, and a couple of database solutions, and a load balancer, and a few other of the services that are trivial to select on your AWS dashboard, then you are going to have to hire somewhere between a dozen and a few dozen highly-qualified people to set them up and run them. (among other reasons, because unless you're Google or Amazon the very few people who can do all of these won't want to work for you) At this point you're talking a payroll of millions of dollars per year.
Add the payroll to the depreciation costs of the building, and you've got your fixed costs; to achieve vaguely Amazon-like efficiency this probably needs to be 25% of your total costs or less. Working it all out, if you've got more than maybe $50 million worth of computers (preferably lots more), and can hire good people, you can run an operation close to as efficiently as Amazon and save lots of money. I would think Bank of America falls in that category. In contrast, for non-huge companies Amazon is a bargain, because you're in effect buying a small fraction of not only a large data center, but of each of the folks who maintain the services you use.