Billing Software Error Sends Billion-Dollar AWS Estimates (theregister.com) 29
AWS says a billing software bug caused some customers to see wildly inflated estimated charges, including reports of accounts showing bills in the billions or even trillions of dollars. The Register reports: An open issue on the AWS Health Dashboard (archived copy at the time of writing) popped up at 1:33 am Pacific time on Friday informing users that Cost Explorer was "reflecting inaccurate estimated billing data." As of writing, the issue is still unresolved despite AWS trying several different things to get it fixed. The company apparently identified the root cause within an hour and a half of beginning its investigation, only describing it as "an issue with unit pricing within the estimated billing computation subsystem."
AWS followed up by pausing estimated bill updates, saying customers would continue to see the inflated figures already displayed, but that those estimates would not increase further. "The displayed billing estimates do not reflect actual usage and charges," AWS explained, noting that customers don't need to take any action, like, we imagine, flooding the help portal with tickets telling them what they already know, for instance.
"Once the issue has been mitigated, we expect full resolution to take multiple hours as we work through recomputing the estimated billing data," AWS added. After we first published this article, Amazon updated the issue page to indicate that it had identified the root cause and mitigated the underlying issue. The company says that it's begun backfilling data in the Cost Management Console to correct billing numbers, and that all customers should see corrected amounts by Saturday, July 18 at noon pacific time.
AWS followed up by pausing estimated bill updates, saying customers would continue to see the inflated figures already displayed, but that those estimates would not increase further. "The displayed billing estimates do not reflect actual usage and charges," AWS explained, noting that customers don't need to take any action, like, we imagine, flooding the help portal with tickets telling them what they already know, for instance.
"Once the issue has been mitigated, we expect full resolution to take multiple hours as we work through recomputing the estimated billing data," AWS added. After we first published this article, Amazon updated the issue page to indicate that it had identified the root cause and mitigated the underlying issue. The company says that it's begun backfilling data in the Cost Management Console to correct billing numbers, and that all customers should see corrected amounts by Saturday, July 18 at noon pacific time.
Amazon Test Run (Score:1)
for hyperinflation, haha
Re: (Score:2)
I'll Bet That... (Score:1)
I'll bet that many bricks were shat.
Dear Management (Score:1)
Just wait... (Score:3, Informative)
...until customer support is entirely staffed by AI agents and the only response you'll get when you contact them about the billion dollar billing error is: "your service will now be terminated because your bill is overdue."
Re: (Score:3)
Well, when your entire business is on someone else's server, this is the risk you run.
Re: (Score:2)
Termination would be a best case scenario. Potentially they could sue you and win because ToS is basically that you have to pay whatever they tell you to.
Happened to me (Score:5, Interesting)
I woke up this morning to an alert that my tiny static web site had incurred $3 million in S3 storage charges. Is that what "going viral" feels like? No thanks.
It was a good reminder to review what protections I have from liability should some script kiddie ddos me.
Re: (Score:3)
Out of morbid curiosity, what does your monthly bill normal run at?
Re: Happened to me (Score:2)
Under $20. My alert threshold has been set to $16 for years.
Re: Happened to me (Score:4, Interesting)
Good thing you didn't have auto-pay enabled...
Trillion dollar estimates (Score:3)
That more than all the wealth in the entire world (~$450tn).
I guess Amazon are trying to outdo Elon....
Re: (Score:2)
It's a cost estimate from AWS Health Dashboard, and with that word in the name the trillion $ amount actually sounds credible.
Gotta love that AI engineering (Score:1)
including reports of accounts showing bills in the billions or even trillions of dollars. The Register reports
I think we are still going to see these kinds of errors making it through on AWS for a while. It's strange in that before AI, correctness seemed to be important for maintaining trust. It's even more strange in that in the competition for incorrect fuzzy answers, it's increasing the price of hardware for everyone. I can't help but think that there is no saving humans.
Is the cost of the Engineers really decreasing so much with AI that it offsets customer losses? Or is this a monopoly type of thing?
Re: (Score:2)
Indeed. There's no doubt that:
- AI wrote the code
- AI wrote the unit tests
- AI did the pull request review
- AI approved all of the above
What could possibly go wrong?
Re: (Score:2)
Is the cost of the Engineers really decreasing so much with AI that it offsets customer losses? Or is this a monopoly type of thing?
The most likely current scenario is that even without all the damage it does, LLM-based coding is significantly more expensive (when you have to pay the real cost) than competent human engineers doing the same job. Of course, if you add all the defects LLM code has into the mix ...
And this is why ... (Score:3)
When the Y2K issues rolled around, a number of people asked me (based on my previous utility experience) whether the lights would stay on. My reply was: "The power company made it through the Year 1900 and not much has changed since then. Sure, you might get a bill for 100 years of consumption. But humans look at this stuff and nobody is going to hold you to that or cut your power."
Unfortunately, humans are increasingly out of the loop.
humans are increasingly out of the loop so take th (Score:2)
humans are increasingly out of the loop so take that lobster for $0.00999 per pound when it should of been $9.99 per pound. my ticket says paid in full.
Far from the worst error that could occur. (Score:4, Insightful)
billing $0.05 per byte vs $0.05 per GB may doit. (Score:2)
billing $0.05 per byte vs $0.05 per GB may doit.
Billing estimates, not actual charges (Score:2)
did someone mess up cents vs dollars on the unit? (Score:2)
did someone mess up cents vs dollars on the unit?
Like should of been 0.000005 cents per KB but was put in as 0.000005 dollars per KB
Re: (Score:2)
Not someone, some AI coding bot.
Re: (Score:2)
This would still not provide an explanation for trillion $ charges ...
Re: (Score:2)
I have always wondered why common computer languages haven't attempted to add units to the base language (note units are distinct from datatypes). I know, F# has it - and probably other minor languages, but no major language has done so. Errors such as incorrect conversions between dollars and cents (or km and miles) would be caught at compile time. Numbers are not just numbers, they almost always have some unit attached to them. Having units would simultaneously improve the readability of the code.
Re: (Score:2)
The algorithm seems to have been failing to scale correctly, yes. On closer inspection, it looks like the billing software treated the bytes used as K used.
I must've put a decimal point in the wrong place (Score:2)
I always do that. I always mess up some mundane detail.
Quick question. (Score:2)
Did AWS use Grok to generate the billing sheets or the code for handling billing?
I'm serious. The errors reported look suspiciously like AI hallucinations or a signed integer being treated as unsigned.