Charter Accidentally Wipes 14K Email Accounts 213
dacut writes with the sad news that Charter Communications, which provides cable and Internet access to 2.6 million customers, accidentally and irretrievably wiped out 14,000 active email accounts while trying to clear out unused accounts. They're providing a $50 credit to each affected customer, which seems a paltry sum for anyone who was less than diligent about backing up their email — though those who relied on Charter's webmail interface had no easy way to accomplish backups. From the article: "There is no way to retrieve the messages, photos and other attachments that were erased from inboxes and archive folders across the country on Monday, said Anita Lamont, a spokeswoman for the suburban St. Louis-based company. 'We really are sincerely sorry for having had this happen and do apologize to all those folks who were affected by the error,' Lamont said Thursday when the company announced the gaffe."
Re:Crap (Score:4, Informative)
Re:Backups (Score:5, Informative)
The backup architecture required to efficiently and safely protect this kind of environment would cost easily several hundred thousand dollars and several full time employees to manage.
Before anyone jumps in with "just buy a bunch of cheap IDE hard drives and rsync, tar, etc...", please don't forget that we're talking about a major server farm, probably in several locations, consisting of likely hundreds (if not thousands) of servers and mail stores.
More than likely, Charter made the business decision that (as other posters have pointed out) email is a volatile storage medium and their internal checks and balances (RAID, etc) were sufficient for protecting against loss. Obviously, they made a mistake and miscalculation. At the end of the day, however, I suspect they'll implement more checks and balances to protect against human error, but I'd be really surprised if IBM/SUN/etc got a big order for a tape library/upgrade. I just can't imagine a company like Charter spending the money (hardware, consumables, people) to back up "Forward this to 10 people in the next 10 minutes and Bill Gates will give you a hand job" messages.
Re:Crap (Score:3, Informative)
Re:Backups (Score:5, Informative)
As opposed to, say, the $700,000k they just paid out.
--Rob
Probably not in the SLA anyway (Score:3, Informative)
Mostly likely their asses were covered by their service agreement. I am pretty sure that Yahoo's policy for lost email is "tough shit" as well.
Re:Crap (Score:3, Informative)
They also restored a lot of the lost data. Something which Charter is completely incapable of.
Yeah, I worked there. (Score:2, Informative)
Re:Backups (Score:5, Informative)
Just because I can, I did a couple of bar-napkin type calculations to see what it would take to protect this environment.
I have no idea what Charter's cable modem subscriber base is, so I took some wild ass guesses. According to Charter's website, they have around 5.7 million customers. Assuming that a 12.5% of them are cable modem subscribers, and each of those accounts has an average of two mailboxes, that gives us just under 1.5 million mailboxes to protect. Further assuming an average mailbox size of 50MB (not unreasonable, given the similar environments I've seen), that's somewhere in the neighborhood of 71 terabytes of data - just for email. That's not counting the supporting infrastructure (authentication, transport, etc).
So to protect 71 terabytes of data, we need somewhere to put that. Tape is most likely. Let's assume LTO3 (probably the most commonly deployed tape technology today in the open systems world), so we've got a raw capacity of 400GB per tape (don't believe the compression specs, I rarely see more than 600GB in the wild). Assuming daily backups kept one week, and weekly backups kept for a month, we'd need about 1780 tapes for the month's rotation. At $40/tape, that's $71,200 in media. Figure 10% per year to replace failed media, and we've got a first year consumable cost of $78,320.
Now, to get the data onto our ~$80k worth of tapes.
Let's figure a 12 hour backup window. (We'll assume that this backup infrastructure will be used to protect some other assets in the other 12 hours) To move 71 terabytes of data in 12 hours, we'll need about 28 LTO3 tape drives (I'll spare you the calculations used to get there - but suffice it to say that I included reasonable overhead and observed real-world performance). At $3k a pop (for quality, supportable, maintainable drives), that's $85k in drives. A tape library to contain said drives will be somewhere in the $100k-$150k range depending on options (redundant robotics, etc). The SAN infrastructure required to connect these drives should be in the $30-$40k range.
So just tape hardware, lets call it $250k.
Additionally, we need backup servers to handle all this data. No, cheap 1U Intel boxes aren't going to cut it. You're going to need some serious iron to drive 28 LTO3 tape drives at full capacity. Off the top of my head, I'd say you're looking at 2-3 mid-sized Unix servers (IBM System p, etc) loaded down with 4GB fibre adapters. Easily another $150k.
Right now, we're just under $500k, and we haven't even started talking about software licensing (Tivoli Storage Manager, Veritas Netbackup, ComVault, etc), infrastructure for the systems being backed up (dedicated Ethernet, or depending on volume, dedicated fibre), miscellaneous supporting infrastructure (power, UPS, air conditioning, etc), and so on.
Once you've got all that, who's going to manage that? Probably a senior backup administrator/architect (90-120k yearly), a mid-range systems administrator (60-90k yearly), and one or two operators (media handling, etc, 30-50k yearly). So that's $250k or so in salaries to manage this beast, figure a benefit load of 60%, and we're at $400k to employ these people.
Initial hardware investment : $480k
Yearly consumables : $8k
Yearly media storage : $60k (no idea - completely made this one up - anyone with knowledge of Iron Mountain, etc, want to comment?)
Yearly salaries to manage: $400k.
Completely ignoring data center costs (AC, power, etc) and software, let's call it an up front investment of $1mil, and a yearly ongoing cost of $500k to support.
14,000 customers at $50 service credit (not real cash) = $700,000
So as long as they only do this once every two years or so, they're in the black.
Sorry, but that's how businesses think. (And yes, there's the cost to customer satisfaction, lost customers, etc, but growth will easily outstrip those losses)
Damn, posting on slashdot feels way too much like real work.
Re:Standard statement... (Score:4, Informative)
I'm not exactly sure what you mean by "entitled to expect."
Gmail is a free (to the user) service. The user might expect it to be reliable, but they are entitled to nothing. If it all goes away, there's no recourse. Also, in light of all the free competing e-mail services these days, most ISP's offer e-mail as a courtesy and will not guarantee it's integrity. It's probabbly carefully worded in every AUP. To practice a business where you are legally responsible for terabytes of customer data is scary can of worms... let Citibank deal with that.
I work in the ISP business and can tell you that the overhead for e-mail is greater then any other service provided. Every other piece of vital data here is peanuts compared to the size of our customer e-mail storage arrays. We back it up as often as we can. It's an absolutely enormous amount of data that changes every second. We do the best we can but, Hell no, I don't want to be liable for it. I would hope you don't expect me to. Even though I know it's being rsyncd off site twice a day I still advise customers every chance I get. Download your messages and archive the important ones periodically.
As the old saying goes, the only person you can rely on is yourself.
Re:"No way"? (Score:3, Informative)
Frightening Words (Score:3, Informative)
I was sharing an office with the lab's sysadmin. One day, while I was happily programming away, I heard the quiet utterance from my office-mate: "Oh, shit." Shivers ran down my spine and I started to panic. I knew immediately that all hell was about to break loose.
Truly a frightening phrase to hear from your sysadmin.
State and Local Govt email archives (Score:3, Informative)
Ever wonder why so many state and local government email system run on Lotus Notes/Domino? It's because Lotus has a built-in feature called "mail journaling" that automatically does the archiving. In addition, Lotus has a standard clustering capability in its design that allows you to replicate the entire servers and their contents effortlessly across multiple machines. When I first had to learn Lotus, I thought it was going to kill me, but the more time I spend with it, the more I realize it is an incredibly powerful and capable messaging and application/database platform. But it has a super-weird learning curve to it that most people never can seem to "get it", hence the widespread fear and loathing towards Lotus Notes.
Re:Crap (Score:3, Informative)
So, even if my Gmail account is lost, my job's IMAP is also lost, I do something really stupid in my home computer and either lose my mboxes here or upload corrupted files to S3, and my house burns down, I'll still have a good chance of recovering most, if not all, of my e-mails.
After losing two or three hard disks I learned to take backups seriously. Good thing it's easier now than when our only reasonably cheap option were 1.44 MB floppies.