Slashdot Log In
Charter Accidentally Wipes 14K Email Accounts
Posted by
kdawson
on Friday January 25, @11:29AM
from the thanks-for-the-memories dept.
from the thanks-for-the-memories dept.
dacut writes with the sad news that Charter Communications, which provides cable and Internet access to 2.6 million customers, accidentally and irretrievably wiped out 14,000 active email accounts while trying to clear out unused accounts. They're providing a $50 credit to each affected customer, which seems a paltry sum for anyone who was less than diligent about backing up their email — though those who relied on Charter's webmail interface had no easy way to accomplish backups. From the article: "There is no way to retrieve the messages, photos and other attachments that were erased from inboxes and archive folders across the country on Monday, said Anita Lamont, a spokeswoman for the suburban St. Louis-based company. 'We really are sincerely sorry for having had this happen and do apologize to all those folks who were affected by the error,' Lamont said Thursday when the company announced the gaffe."
Related Stories
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
my gut feelings.... (Score:5, Funny)
You just know this must be related to the story IT: You Used Perl to Write WHAT?! [slashdot.org] from earlier this morning...
Re: (Score:3, Funny)
Re: (Score:3, Insightful)
You know, the corporate version of the celebrity arrest...
Re:my gut feelings.... (Score:5, Funny)
The irony? Slashdot dove into 503 and 500 errors a few minutes after you posted that.
Crap (Score:3, Interesting)
Moral: Exercise! (Score:2)
test it, test it, test it
and over, and over, and over
again and again and again
and back-up and back-up and back-up...
feel the burn!
Re:Crap (Score:5, Interesting)
Which is part of the reason why I actually trust Google with my email. I wouldn't mind them providing a proper backup mechanism (no, POP3 isn't a worthwhile mechanism for me), but it simply isn't as necessary as some hosting providers.
That being said, this entire mess could have been avoided if someone took a tape backup before purging data from the system...
Re:Crap (Score:4, Informative)
Re: (Score:2)
Re: (Score:3, Informative)
Re: (Score:3, Interesting)
Re:Crap (Score:5, Funny)
There's always Google paper [google.com]...
Re: (Score:3, Informative)
They also restored a lot of the lost data. Something which Charter is completely incapable of.
Re:Crap (Score:5, Insightful)
One might be able to reach a person at Charter, but a helpful person? Not on your life. You speak about behemoth corporations, and Charter embodies the worst of corporate bureaucracy. They are total idiots, the left hand doesn't talk to the right hand, and their prices are unreasonable. And yes, I dumped them as soon as I could so I don't have to deal with them any more. But not once did I deal with a helpful person.
And deleting 14,000 email accounts just shows the heights of stupidity this company has achieved.
Re:Crap (Score:5, Interesting)
A year or two ago, our internet access stopped working. Over the course of three weeks, I spent 12 hours on the phone with Charter support. During those calls I was told: they had deleted my MAC address from the database, that they had no record of me ever being a Charter customer (despite the cable boxes in my living room), that they had no record of me having internet access, and that I was stealing cable.
Finally, I got fed up, and called AT&T, got DSL and Dish. Once everything was working, I packed up all of Charter's equipment, brought it down to a local office, and told them to shut me off. This was a Saturday.
On Monday, my Dish stops working. I call up the Dish people; they come out around 10 days later to tell me that Charter had disconnected my satellite and had put a terminator on the line.
Fuck Charter. Fuck them in the neck with a donkey cock. They are the worst of the worst of companies (believe it or not, this is the short version - the long version is worse). If they actually had to compete, instead of having a monopoly, they would have been out of business years ago - and I would have bought some of their assets so I could set them on fire.
Fuck Charter.
Re: (Score:3, Informative)
Email? (Score:5, Funny)
Standard statement... (Score:5, Interesting)
Yes, they should have had backups now days, but none the less, if you want it saved, don't leave it in your inbox.
I've had folks complain that the trash automatically was cleaned out every three days. WTF?
Re:Standard statement... (Score:5, Insightful)
I see no reason why bytes are any more "volatile" in an IMAP file than anywhere else.
Re:Standard statement... (Score:4, Informative)
I'm not exactly sure what you mean by "entitled to expect."
Gmail is a free (to the user) service. The user might expect it to be reliable, but they are entitled to nothing. If it all goes away, there's no recourse. Also, in light of all the free competing e-mail services these days, most ISP's offer e-mail as a courtesy and will not guarantee it's integrity. It's probabbly carefully worded in every AUP. To practice a business where you are legally responsible for terabytes of customer data is scary can of worms... let Citibank deal with that.
I work in the ISP business and can tell you that the overhead for e-mail is greater then any other service provided. Every other piece of vital data here is peanuts compared to the size of our customer e-mail storage arrays. We back it up as often as we can. It's an absolutely enormous amount of data that changes every second. We do the best we can but, Hell no, I don't want to be liable for it. I would hope you don't expect me to. Even though I know it's being rsyncd off site twice a day I still advise customers every chance I get. Download your messages and archive the important ones periodically.
As the old saying goes, the only person you can rely on is yourself.
Re: (Score:3, Insightful)
It would be impossible to cent
It's like snapshots never happened, isn't it? (Score:3, Insightful)
funny story....(totally true) (Score:4, Funny)
Your statement about people complaining about the trash(deleted email) being emptied brought back a fond memory of mine. I was early in my IT Admin career (was a programmer for over 10 prior to that), I was working for a major insurance company administrating their CC:Mail network. We had issues of people never emptying their "trash" folder and it was taking a long time to do anything for everyone on the system. We sent out a memo to the entire company telling them that in a week we would start emptying the trash folder nightly around midnight before the backups and other maintenance begins. very few people bothered to pay attention to this memo.
Here is where it get...funny. A high-muckety-muck (eg: pointy haired VP) called the help desk screaming for his trash folder to be restored as it was emptied without his permission and had important files in there. He wanted to see the guilty parties in his office post-haste. I was part of the team, so I had to go to his office. while waiting outside his door for him to let our team in, I grabbed a trashcan and some vertical file folders and paper from his secretary. (can you see where I'm going yet?) We were let in, and he proceeded to rip us up one side and down the other. Our manager brought a copy of the memo, which he promptly threw away while continuing to yell.
I asked to speak (everyone else was quietly taking the heat) and proceeded to put the trash can on the desk, put the vertical file folders in the trashcan and put paper in each folder. While I was explaining this analogy to him, I asked if every day his office trashcan was empty when he came into the office. He said yes. I basically gave him the analogy that the trash folder in cc:mail was the same as his office trashcan - whatever went in during the day was retrievable, but at midnight, the office cleaning crew would come in and empty his trashcan and we also would be emptying his electronic trash folder the same way. He blinked, and understood what we were doing, pulled the memo out of the trash and reread it. we were dismissed with no apology back to our offices.
Note that I was a contractor - I couldn't be fired...just sent away back to my firm to go on to the next assignment. I ended up being there for 2.5 years. I still laugh at that situation even today.
Re: (Score:3, Insightful)
State and Local Govt email archives (Score:3, Informative)
Unfortunate (Score:4, Insightful)
It's one thing if you have angry customers over something you have control over. It's another thing entirely if your customers are angry at you AND there isn't a single solitary thing you can do. That said, I hope that they are more careful in the future...
Probably not in the SLA anyway (Score:3, Informative)
"No way"? (Score:3, Insightful)
Re: (Score:3, Informative)
Things you dont want to hear from your DBA.. (Score:3, Funny)
/golfclap (Score:5, Funny)
Charter Employee: Yessiree ! 'Click'
Charter Employee: Oh shit.
Charter Manager: What ?
Charter Employee: 'Surfs over to Monster.com' Oh nothing. Nothing at all.
Frightening Words (Score:3, Informative)
I was sharing an office with the lab's sysadmin. One day, while I was happily programming away, I
Backups (Score:2)
Re:Backups (Score:5, Informative)
The backup architecture required to efficiently and safely protect this kind of environment would cost easily several hundred thousand dollars and several full time employees to manage.
Before anyone jumps in with "just buy a bunch of cheap IDE hard drives and rsync, tar, etc...", please don't forget that we're talking about a major server farm, probably in several locations, consisting of likely hundreds (if not thousands) of servers and mail stores.
More than likely, Charter made the business decision that (as other posters have pointed out) email is a volatile storage medium and their internal checks and balances (RAID, etc) were sufficient for protecting against loss. Obviously, they made a mistake and miscalculation. At the end of the day, however, I suspect they'll implement more checks and balances to protect against human error, but I'd be really surprised if IBM/SUN/etc got a big order for a tape library/upgrade. I just can't imagine a company like Charter spending the money (hardware, consumables, people) to back up "Forward this to 10 people in the next 10 minutes and Bill Gates will give you a hand job" messages.
Re:Backups (Score:5, Informative)
As opposed to, say, the $700,000k they just paid out.
--Rob
Re:Backups (Score:5, Informative)
Just because I can, I did a couple of bar-napkin type calculations to see what it would take to protect this environment.
I have no idea what Charter's cable modem subscriber base is, so I took some wild ass guesses. According to Charter's website, they have around 5.7 million customers. Assuming that a 12.5% of them are cable modem subscribers, and each of those accounts has an average of two mailboxes, that gives us just under 1.5 million mailboxes to protect. Further assuming an average mailbox size of 50MB (not unreasonable, given the similar environments I've seen), that's somewhere in the neighborhood of 71 terabytes of data - just for email. That's not counting the supporting infrastructure (authentication, transport, etc).
So to protect 71 terabytes of data, we need somewhere to put that. Tape is most likely. Let's assume LTO3 (probably the most commonly deployed tape technology today in the open systems world), so we've got a raw capacity of 400GB per tape (don't believe the compression specs, I rarely see more than 600GB in the wild). Assuming daily backups kept one week, and weekly backups kept for a month, we'd need about 1780 tapes for the month's rotation. At $40/tape, that's $71,200 in media. Figure 10% per year to replace failed media, and we've got a first year consumable cost of $78,320.
Now, to get the data onto our ~$80k worth of tapes.
Let's figure a 12 hour backup window. (We'll assume that this backup infrastructure will be used to protect some other assets in the other 12 hours) To move 71 terabytes of data in 12 hours, we'll need about 28 LTO3 tape drives (I'll spare you the calculations used to get there - but suffice it to say that I included reasonable overhead and observed real-world performance). At $3k a pop (for quality, supportable, maintainable drives), that's $85k in drives. A tape library to contain said drives will be somewhere in the $100k-$150k range depending on options (redundant robotics, etc). The SAN infrastructure required to connect these drives should be in the $30-$40k range.
So just tape hardware, lets call it $250k.
Additionally, we need backup servers to handle all this data. No, cheap 1U Intel boxes aren't going to cut it. You're going to need some serious iron to drive 28 LTO3 tape drives at full capacity. Off the top of my head, I'd say you're looking at 2-3 mid-sized Unix servers (IBM System p, etc) loaded down with 4GB fibre adapters. Easily another $150k.
Right now, we're just under $500k, and we haven't even started talking about software licensing (Tivoli Storage Manager, Veritas Netbackup, ComVault, etc), infrastructure for the systems being backed up (dedicated Ethernet, or depending on volume, dedicated fibre), miscellaneous supporting infrastructure (power, UPS, air conditioning, etc), and so on.
Once you've got all that, who's going to manage that? Probably a senior backup administrator/architect (90-120k yearly), a mid-range systems administrator (60-90k yearly), and one or two operators (media handling, etc, 30-50k yearly). So that's $250k or so in salaries to manage this beast, figure a benefit load of 60%, and we're at $400k to employ these people.
Initial hardware investment : $480k
Yearly consumables : $8k
Yearly media storage : $60k (no idea - completely made this one up - anyone with knowledge of Iron Mountain, etc, want to comment?)
Yearly salaries to manage: $400k.
Completely ignoring data center costs (AC, power, etc) and software, let's call it an up front investment of $1mil, and a yearly ongoing cost of $500k to support.
14,000 customers at $50 service credit (not real cash) = $700,000
So as long as they only do this once every two years or so, they're in the black.
Sorry, but that's how businesses think. (And yes, there's the cost to customer satisfaction, lost customers, etc, but growth will easily outstrip those losses)
Damn, posting on slashdot feels way too much like real work.
Re: (Score:3, Funny)
Thunderbird (Score:2)
No backups? Come on! (Score:4, Insightful)
So, either Charter doesn't back up email very well, or their process to "clear out old accounts" involves actually deleting all of the backups of those accounts as well. I already addressed the issue with the former scenario, but if it's the latter, I'd have to say that's a pretty nasty practice too. Any time you clear out old and "unused" data, you have to assume that you're likely to accidentally hit some false positives, which is one of the reasons we have backups in the first place.
Transcript of the incident (Score:5, Funny)
clickety clickety (SIGH) clickety (beep)
clickety clickety (beep)
clickety (beep)
clickety clickety (beep)
click- OHHHH SH**! F***!
Re:Transcript of the incident (Score:4, Funny)
Boss: "We need more space on the server"
System Operator: (clickety, clickety) "Sure! What's your username again?"
Boss: (pauses) "Oh shit..."
Re:Transcript of the incident (Score:4, Funny)
The case against remotely hosted applications (Score:3, Insightful)
I always POP my email down to my own local computer.
At least if
Don't let your access provider do your email (Score:2)
No excuse for this (Score:2)
My e-mail, on my computer (Score:2)
Host you own (Score:5, Interesting)
It isn't hard, either. My 63-yo father is now doing the same thing, as he switched ISPs for the first time now that he can get DSL out on the farm, and he isn't the most technical guy.
Email not backed up (Score:4, Interesting)
That said, it's standard practice when deleting an account to mark the data as deleted, so that it looks like it's gone to the user, but it's actually pending deletion later. Then, when someone complains or pays their bill, you can restore what was "deleted." After a predetermined amount of time, if you don't complain, a cleanup script deletes it permenently.
poop happens (Score:3, Insightful)
But what's done is done and props to them for a bullshit-free apology.
Most people are prepared to cut you some slack when you screw up as long as you admit your mistake.
- recognise what it was that you did wrong
- claim responsibility for your actions
- apologise
- state clearly what you learned and what actions you will take to prevent a recurrence
Or you could take to legal advice / bush administration route
- flatly refuse to acknowledge that anything bad actually happened
- talk about how 'the other guys' screw up all the time
- start an internal investigation and refuse to comment on the issue while it is under investigation
- eventually admit that 'mistakes were made' but no, you can't think of any specific examples right now and it was all someone else's fault and you there's no way you could have known it would happen.
Spokeman talk (Score:3, Insightful)
Note the use of the passive voice, which is commonly done to avoid taking responsibility. It seems like even when they're trying to apologize, spin-doctors can't turn off their instinct of avoiding responsibility for mistakes.