Charter Accidentally Wipes 14K Email Accounts 213
dacut writes with the sad news that Charter Communications, which provides cable and Internet access to 2.6 million customers, accidentally and irretrievably wiped out 14,000 active email accounts while trying to clear out unused accounts. They're providing a $50 credit to each affected customer, which seems a paltry sum for anyone who was less than diligent about backing up their email — though those who relied on Charter's webmail interface had no easy way to accomplish backups. From the article: "There is no way to retrieve the messages, photos and other attachments that were erased from inboxes and archive folders across the country on Monday, said Anita Lamont, a spokeswoman for the suburban St. Louis-based company. 'We really are sincerely sorry for having had this happen and do apologize to all those folks who were affected by the error,' Lamont said Thursday when the company announced the gaffe."
my gut feelings.... (Score:5, Funny)
You just know this must be related to the story IT: You Used Perl to Write WHAT?! [slashdot.org] from earlier this morning...
Re: (Score:3, Funny)
Re: (Score:3, Insightful)
You know, the corporate version of the celebrity arrest...
Re:my gut feelings.... (Score:5, Funny)
The irony? Slashdot dove into 503 and 500 errors a few minutes after you posted that.
Crap (Score:3, Interesting)
Moral: Exercise! (Score:2)
test it, test it, test it
and over, and over, and over
again and again and again
and back-up and back-up and back-up...
feel the burn!
Re:Crap (Score:5, Interesting)
Which is part of the reason why I actually trust Google with my email. I wouldn't mind them providing a proper backup mechanism (no, POP3 isn't a worthwhile mechanism for me), but it simply isn't as necessary as some hosting providers.
That being said, this entire mess could have been avoided if someone took a tape backup before purging data from the system...
Re:Crap (Score:4, Informative)
Re: (Score:2)
Re: (Score:3, Informative)
Re: (Score:2)
Why do you care about time for a backup, though? Not criticizing, just wondering.
Re: (Score:3, Interesting)
Re:Crap (Score:5, Funny)
There's always Google paper [google.com]...
Re: (Score:2)
Re:Crap (Score:5, Insightful)
One might be able to reach a person at Charter, but a helpful person? Not on your life. You speak about behemoth corporations, and Charter embodies the worst of corporate bureaucracy. They are total idiots, the left hand doesn't talk to the right hand, and their prices are unreasonable. And yes, I dumped them as soon as I could so I don't have to deal with them any more. But not once did I deal with a helpful person.
And deleting 14,000 email accounts just shows the heights of stupidity this company has achieved.
Re:Crap (Score:5, Interesting)
A year or two ago, our internet access stopped working. Over the course of three weeks, I spent 12 hours on the phone with Charter support. During those calls I was told: they had deleted my MAC address from the database, that they had no record of me ever being a Charter customer (despite the cable boxes in my living room), that they had no record of me having internet access, and that I was stealing cable.
Finally, I got fed up, and called AT&T, got DSL and Dish. Once everything was working, I packed up all of Charter's equipment, brought it down to a local office, and told them to shut me off. This was a Saturday.
On Monday, my Dish stops working. I call up the Dish people; they come out around 10 days later to tell me that Charter had disconnected my satellite and had put a terminator on the line.
Fuck Charter. Fuck them in the neck with a donkey cock. They are the worst of the worst of companies (believe it or not, this is the short version - the long version is worse). If they actually had to compete, instead of having a monopoly, they would have been out of business years ago - and I would have bought some of their assets so I could set them on fire.
Fuck Charter.
Re: (Score:2)
And yet they also lost a bunch of accounts a couple of years ago, albeit far fewer than in this case.
Re: (Score:3, Informative)
They also restored a lot of the lost data. Something which Charter is completely incapable of.
Re: (Score:3, Informative)
What I do is this: I use Gmail as my main e-mail address, but I have it set up in IMAP mode at my job's Thunderbird. In my home, I also have Thunderbird downloading e-mails, but in POP3 mode. Why, you ask? Simple: because there I have many special filters set to distribute my mail to special mailboxes so as to make it easy, and fast, to backup them, in encrypted form, to Amazon S3 [amazon.com] using Jungle Disk [jungledisk.com] (together wi
Email? (Score:5, Funny)
Standard statement... (Score:5, Interesting)
Yes, they should have had backups now days, but none the less, if you want it saved, don't leave it in your inbox.
I've had folks complain that the trash automatically was cleaned out every three days. WTF?
Re:Standard statement... (Score:5, Insightful)
I see no reason why bytes are any more "volatile" in an IMAP file than anywhere else.
Re: (Score:2)
Re: (Score:2)
Re:Standard statement... (Score:4, Informative)
I'm not exactly sure what you mean by "entitled to expect."
Gmail is a free (to the user) service. The user might expect it to be reliable, but they are entitled to nothing. If it all goes away, there's no recourse. Also, in light of all the free competing e-mail services these days, most ISP's offer e-mail as a courtesy and will not guarantee it's integrity. It's probabbly carefully worded in every AUP. To practice a business where you are legally responsible for terabytes of customer data is scary can of worms... let Citibank deal with that.
I work in the ISP business and can tell you that the overhead for e-mail is greater then any other service provided. Every other piece of vital data here is peanuts compared to the size of our customer e-mail storage arrays. We back it up as often as we can. It's an absolutely enormous amount of data that changes every second. We do the best we can but, Hell no, I don't want to be liable for it. I would hope you don't expect me to. Even though I know it's being rsyncd off site twice a day I still advise customers every chance I get. Download your messages and archive the important ones periodically.
As the old saying goes, the only person you can rely on is yourself.
Re: (Score:2)
I mean it in the context of the original comment: third parties engaging in finger pointing should be pointing at the goofballs who wiped out mailboxes without having a proper backup in place -- not claiming that email is some sort of tempfile that users should expect to disappear without warning.
Legal recourse? IANAL, and I haven't seen the agreement the users accepted, but my guess is that they got $50 more than they're "entitled to" in that sens
Re: (Score:3, Insightful)
I have some empathy for these goofballs. I'd say that that many accounts was probabbly about 60GB of data. 60GB out of God knows how many terabytes.
It would be impossible to centralize that much data, except for perhaps the database to verify users. Mbox data would have to be distributed. That would eliminate any single point of failure, but also increase the chance of a small failure. Say you have 60 servers, and 6,000 disks. Say
It's like snapshots never happened, isn't it? (Score:3, Insightful)
Re: (Score:2)
Kinda funny though, our in-house lawyer wiped out his entire inbox (which is all stored server-side) about 2 years ago. I almost chuckled when he asked why we didn't keep a backup and I had to remind him that HE suggested that we not do so for legal reasons . .
We actually had a
Re: (Score:2)
Re: (Score:2)
If say, we were involved in a sexual harassment case. A few harmless dirty jokes passed between coworkers generally doesn't cause any harm (talking about willing parties
Re: (Score:2)
The onus for archiving was on the user, but if you didn't archive it you lost it after a year (90 days) if it was in your inbox.
As the sibling poster points out, there are a lot of very specific document rete
State and Local Govt email archives (Score:3, Informative)
Re:email servers are not long term archives... (Score:2, Insightful)
1. Most access is plain text and subject to snooping/hijacking (passwd/userid/content)
2. Email is the most abused internet protocol (my opinion) by zombots, spammers, and virus/trojan propagators. ISPs do a lot to counter spam and threatening content but sometimes they get hosed. Or your home machine
funny story....(totally true) (Score:4, Funny)
Your statement about people complaining about the trash(deleted email) being emptied brought back a fond memory of mine. I was early in my IT Admin career (was a programmer for over 10 prior to that), I was working for a major insurance company administrating their CC:Mail network. We had issues of people never emptying their "trash" folder and it was taking a long time to do anything for everyone on the system. We sent out a memo to the entire company telling them that in a week we would start emptying the trash folder nightly around midnight before the backups and other maintenance begins. very few people bothered to pay attention to this memo.
Here is where it get...funny. A high-muckety-muck (eg: pointy haired VP) called the help desk screaming for his trash folder to be restored as it was emptied without his permission and had important files in there. He wanted to see the guilty parties in his office post-haste. I was part of the team, so I had to go to his office. while waiting outside his door for him to let our team in, I grabbed a trashcan and some vertical file folders and paper from his secretary. (can you see where I'm going yet?) We were let in, and he proceeded to rip us up one side and down the other. Our manager brought a copy of the memo, which he promptly threw away while continuing to yell.
I asked to speak (everyone else was quietly taking the heat) and proceeded to put the trash can on the desk, put the vertical file folders in the trashcan and put paper in each folder. While I was explaining this analogy to him, I asked if every day his office trashcan was empty when he came into the office. He said yes. I basically gave him the analogy that the trash folder in cc:mail was the same as his office trashcan - whatever went in during the day was retrievable, but at midnight, the office cleaning crew would come in and empty his trashcan and we also would be emptying his electronic trash folder the same way. He blinked, and understood what we were doing, pulled the memo out of the trash and reread it. we were dismissed with no apology back to our offices.
Note that I was a contractor - I couldn't be fired...just sent away back to my firm to go on to the next assignment. I ended up being there for 2.5 years. I still laugh at that situation even today.
Re: (Score:3, Insightful)
Unfortunate (Score:4, Insightful)
It's one thing if you have angry customers over something you have control over. It's another thing entirely if your customers are angry at you AND there isn't a single solitary thing you can do. That said, I hope that they are more careful in the future...
Re: (Score:2)
There's actually legislation at the state level right now to attempt to loosen their stranglehold of poor service.
Re: (Score:2)
But this is why you're supposed to be careful to avoid screwing things up this seriously. I appreciate that you're trying to take some kind of sympathetic angle here but I don't see the relevance.
There were a variety of things that a company with 2.6
Probably not in the SLA anyway (Score:3, Informative)
Mostly likely their asses were covered by their service agreement. I am pretty sure that Yahoo's policy for lost email is "tough shit" as well.
Re: (Score:2)
"No way"? (Score:3, Insightful)
Re: (Score:2)
Re: (Score:3, Informative)
Re: (Score:2)
Re: (Score:2)
No
Re: (Score:2)
This is Charter. Charter customers are just happy they don't come to your house every day and kick you in the balls.
Things you dont want to hear from your DBA.. (Score:3, Funny)
Re: (Score:2)
Re: (Score:2)
Oh, I don't know about that. I've worked with four DBAs in person, and I wouldn't mind at all hearing that from Lucy. Nor would I be terribly surprised.
I'll pass on the other three, though. Especially the dead one.
/golfclap (Score:5, Funny)
Charter Employee: Yessiree ! 'Click'
Charter Employee: Oh shit.
Charter Manager: What ?
Charter Employee: 'Surfs over to Monster.com' Oh nothing. Nothing at all.
Frightening Words (Score:3, Informative)
I was sharing an office with the lab's sysadmin. One day, while I was happily programming away, I heard the quiet utterance from my office-mate: "Oh, shit." Shivers ran down my spine and I started to panic. I knew immediately that all hell was about to break loose.
Truly a frightening phrase to hear from your sysadmin.
Re: (Score:2)
Charter Manager: These are the correct accounts to nuke.
Charter Employee: Are you sure that these are the correct accounts to nuke?
Charter Manager: I've told you once, just do it.
Later -
Charter Manager: You nuked the wrong accounts, you'd better start looking at monster.com
Re: (Score:2)
Backups (Score:2)
Re:Backups (Score:5, Informative)
The backup architecture required to efficiently and safely protect this kind of environment would cost easily several hundred thousand dollars and several full time employees to manage.
Before anyone jumps in with "just buy a bunch of cheap IDE hard drives and rsync, tar, etc...", please don't forget that we're talking about a major server farm, probably in several locations, consisting of likely hundreds (if not thousands) of servers and mail stores.
More than likely, Charter made the business decision that (as other posters have pointed out) email is a volatile storage medium and their internal checks and balances (RAID, etc) were sufficient for protecting against loss. Obviously, they made a mistake and miscalculation. At the end of the day, however, I suspect they'll implement more checks and balances to protect against human error, but I'd be really surprised if IBM/SUN/etc got a big order for a tape library/upgrade. I just can't imagine a company like Charter spending the money (hardware, consumables, people) to back up "Forward this to 10 people in the next 10 minutes and Bill Gates will give you a hand job" messages.
One problem (Score:2)
I think it's 100% reasonable to expect your e-mail (which is paid for as part of your service) be kept safe and not deleted. I do not think that is asking too much -- if your provider includes that service as part of their product offering.
If Charter had said "hey guys, we'll give you free e-mail but we aren't going to suppor
Re:Backups (Score:5, Informative)
As opposed to, say, the $700,000k they just paid out.
--Rob
Re: (Score:2)
A new car built by my company leaves somewhere traveling at 60 mph. The rear differential locks up. The car crashes and burns with everyone trapped inside. Now, should we initiate a recall? Take the number of vehicles in the field, A, multiply by the probable rate of failure, B, multiply by the average out-of-court settlement, C. A times B times C equals X. If X is less than the cost of a recall, we don't do one.
Re:Backups (Score:5, Informative)
Just because I can, I did a couple of bar-napkin type calculations to see what it would take to protect this environment.
I have no idea what Charter's cable modem subscriber base is, so I took some wild ass guesses. According to Charter's website, they have around 5.7 million customers. Assuming that a 12.5% of them are cable modem subscribers, and each of those accounts has an average of two mailboxes, that gives us just under 1.5 million mailboxes to protect. Further assuming an average mailbox size of 50MB (not unreasonable, given the similar environments I've seen), that's somewhere in the neighborhood of 71 terabytes of data - just for email. That's not counting the supporting infrastructure (authentication, transport, etc).
So to protect 71 terabytes of data, we need somewhere to put that. Tape is most likely. Let's assume LTO3 (probably the most commonly deployed tape technology today in the open systems world), so we've got a raw capacity of 400GB per tape (don't believe the compression specs, I rarely see more than 600GB in the wild). Assuming daily backups kept one week, and weekly backups kept for a month, we'd need about 1780 tapes for the month's rotation. At $40/tape, that's $71,200 in media. Figure 10% per year to replace failed media, and we've got a first year consumable cost of $78,320.
Now, to get the data onto our ~$80k worth of tapes.
Let's figure a 12 hour backup window. (We'll assume that this backup infrastructure will be used to protect some other assets in the other 12 hours) To move 71 terabytes of data in 12 hours, we'll need about 28 LTO3 tape drives (I'll spare you the calculations used to get there - but suffice it to say that I included reasonable overhead and observed real-world performance). At $3k a pop (for quality, supportable, maintainable drives), that's $85k in drives. A tape library to contain said drives will be somewhere in the $100k-$150k range depending on options (redundant robotics, etc). The SAN infrastructure required to connect these drives should be in the $30-$40k range.
So just tape hardware, lets call it $250k.
Additionally, we need backup servers to handle all this data. No, cheap 1U Intel boxes aren't going to cut it. You're going to need some serious iron to drive 28 LTO3 tape drives at full capacity. Off the top of my head, I'd say you're looking at 2-3 mid-sized Unix servers (IBM System p, etc) loaded down with 4GB fibre adapters. Easily another $150k.
Right now, we're just under $500k, and we haven't even started talking about software licensing (Tivoli Storage Manager, Veritas Netbackup, ComVault, etc), infrastructure for the systems being backed up (dedicated Ethernet, or depending on volume, dedicated fibre), miscellaneous supporting infrastructure (power, UPS, air conditioning, etc), and so on.
Once you've got all that, who's going to manage that? Probably a senior backup administrator/architect (90-120k yearly), a mid-range systems administrator (60-90k yearly), and one or two operators (media handling, etc, 30-50k yearly). So that's $250k or so in salaries to manage this beast, figure a benefit load of 60%, and we're at $400k to employ these people.
Initial hardware investment : $480k
Yearly consumables : $8k
Yearly media storage : $60k (no idea - completely made this one up - anyone with knowledge of Iron Mountain, etc, want to comment?)
Yearly salaries to manage: $400k.
Completely ignoring data center costs (AC, power, etc) and software, let's call it an up front investment of $1mil, and a yearly ongoing cost of $500k to support.
14,000 customers at $50 service credit (not real cash) = $700,000
So as long as they only do this once every two years or so, they're in the black.
Sorry, but that's how businesses think. (And yes, there's the cost to customer satisfaction, lost customers, etc, but growth will easily outstrip those losses)
Damn, posting on slashdot feels way too much like real work.
Re: (Score:2)
Keep it simple, cupcake.
... assuming an average mailbox size of 50MB ...
Sounds reasonable. A 1 TB [newegg.com] drive costs $240 (don't forget to send in for that rebate). That works out to roughly:
$240/TB / 1000GB/TB / 1000GB/MB * 50MB = 1.2 pennies
Of course, there are some additional costs involved, but it doesn't cost a heck of a lot per user to back up the email. The more customers, the more cost, but also the more revenue.
Re: (Score:3, Funny)
Thunderbird (Score:2)
Smarter solution yet: (Score:2)
email will be backed up on: Charter's servers, for as long as they can manage. Gmails servers, til judgment day, and thunderbird's mbx storage file, until you have a HDD fail or similar bad luck.
I am often asked to 'set people up on the net' and that is how I always configure their ISP email accounts.. it gives a little bit of extra resilience and also, free virus/spam filtering
No backups? Come on! (Score:4, Insightful)
So, either Charter doesn't back up email very well, or their process to "clear out old accounts" involves actually deleting all of the backups of those accounts as well. I already addressed the issue with the former scenario, but if it's the latter, I'd have to say that's a pretty nasty practice too. Any time you clear out old and "unused" data, you have to assume that you're likely to accidentally hit some false positives, which is one of the reasons we have backups in the first place.
Re: (Score:2)
Re: (Score:2)
I'm sure Google would welcome the traffic and customers, and the ISP saves a bunch on $50 payouts. (of course the ISP would need to make it clear that the mail is with Google etc, and limit their liability should Google delete your mail account.)
Transcript of the incident (Score:5, Funny)
clickety clickety (SIGH) clickety (beep)
clickety clickety (beep)
clickety (beep)
clickety clickety (beep)
click- OHHHH SH**! F***!
Re:Transcript of the incident (Score:4, Funny)
Boss: "We need more space on the server"
System Operator: (clickety, clickety) "Sure! What's your username again?"
Boss: (pauses) "Oh shit..."
Re:Transcript of the incident (Score:4, Funny)
Re: (Score:2)
The case against remotely hosted applications (Score:3, Insightful)
I always POP my email down to my own local computer.
At least if
Don't let your access provider do your email (Score:2)
For one thing, they involve a certain sort of lock-in. For another, the ASP never do the jobs as well as dedicated email or hosting companies.
Get your own domain name, so that you can switch providers (hosting and email) if you need to. M
No excuse for this (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
That said, in that environment, they should never be deleting, but instead marking stuff for
My e-mail, on my computer (Score:2)
And of course, if your Internet connection goes down you're cut off from everything.
Still, I can't believe that the ISP doesn't have a backup somewhere.
The Importance of Backups (Score:2)
Host you own (Score:5, Interesting)
It isn't hard, either. My 63-yo father is now doing the same thing, as he switched ISPs for the first time now that he can get DSL out on the farm, and he isn't the most technical guy.
most companies don't use best practices (Score:2)
Re: (Score:2)
It may always seem that way, but fortunate for the rest of us in IT, it's not always the case. What you don't hear about, of course, are all the times when an organization has deleted critical data by accident, but was able to restore because they did the proper thing and had good backups of the data. That never seems to make headlines, for some reason.
Email not backed up (Score:4, Interesting)
That said, it's standard practice when deleting an account to mark the data as deleted, so that it looks like it's gone to the user, but it's actually pending deletion later. Then, when someone complains or pays their bill, you can restore what was "deleted." After a predetermined amount of time, if you don't complain, a cleanup script deletes it permenently.
rebate? (Score:2)
I had already planned to cancel next Monday when my DSL gets hooked up.
Re: (Score:2)
Just don't pay your last bill until the refund shows up.
Re: (Score:2)
Wow, they finally cleared my "unused" account (Score:2)
Its Charter this is not surprising (Score:2)
wooops (Score:2)
Store on Server? (Score:2, Insightful)
poop happens (Score:3, Insightful)
But what's done is done and props to them for a bullshit-free apology.
Most people are prepared to cut you some slack when you screw up as long as you admit your mistake.
- recognise what it was that you did wrong
- claim responsibility for your actions
- apologise
- state clearly what you learned and what actions you will take to prevent a recurrence
Or you could take to legal advice / bush administration route
- flatly refuse to acknowledge that anything bad actually happened
- talk about how 'the other guys' screw up all the time
- start an internal investigation and refuse to comment on the issue while it is under investigation
- eventually admit that 'mistakes were made' but no, you can't think of any specific examples right now and it was all someone else's fault and you there's no way you could have known it would happen.
Spokeman talk (Score:3, Insightful)
Note the use of the passive voice, which is commonly done to avoid taking responsibility. It seems like even when they're trying to apologize, spin-doctors can't turn off their instinct of avoiding responsibility for mistakes.
If all else fails... (Score:2, Funny)
some days, sysadmining just sux (Score:2, Insightful)
for eight months, i worked for a small-town isp with dsl and dialup customers. we had old equipment and no budget for upgrades. we had an autoloader that would occasionally snap tapes, old drive arrays that would fail with no replacement parts on hand ("whuddya mean, we got harddrives right there" "those are ide, i need scsi3"). backupexec would report completed jobs but find no restorable data
In related news... (Score:2)
Money badly spent (Score:2)
Yeah, I worked there. (Score:2, Informative)
Re:Less than diligent about backing up their email (Score:2)
Re: (Score:2)
Re: (Score:2)