Follow Slashdot blog updates by subscribing to our blog RSS feed

 



Forgot your password?
typodupeerror
Bug Communications

Charter Accidentally Wipes 14K Email Accounts 213

Posted by kdawson
from the thanks-for-the-memories dept.
dacut writes with the sad news that Charter Communications, which provides cable and Internet access to 2.6 million customers, accidentally and irretrievably wiped out 14,000 active email accounts while trying to clear out unused accounts. They're providing a $50 credit to each affected customer, which seems a paltry sum for anyone who was less than diligent about backing up their email — though those who relied on Charter's webmail interface had no easy way to accomplish backups. From the article: "There is no way to retrieve the messages, photos and other attachments that were erased from inboxes and archive folders across the country on Monday, said Anita Lamont, a spokeswoman for the suburban St. Louis-based company. 'We really are sincerely sorry for having had this happen and do apologize to all those folks who were affected by the error,' Lamont said Thursday when the company announced the gaffe."
This discussion has been archived. No new comments can be posted.

Charter Accidentally Wipes 14K Email Accounts

Comments Filter:
  • by grub (11606) <slashdot@grub.net> on Friday January 25, 2008 @12:30PM (#22182504) Homepage Journal

    You just know this must be related to the story IT: You Used Perl to Write WHAT?! [slashdot.org] from earlier this morning...

  • Crap (Score:3, Interesting)

    by Stevyn (691306) on Friday January 25, 2008 @12:31PM (#22182512)
    I am one of those people who uses Gmail as a backup betting it's more reliable than my hard drive.
    • OK, one more time...

      test it, test it, test it
      and over, and over, and over
      again and again and again
      and back-up and back-up and back-up...

      feel the burn!

    • Re:Crap (Score:5, Interesting)

      by AKAImBatman (238306) <akaimbatman@gmailMOSCOW.com minus city> on Friday January 25, 2008 @12:38PM (#22182614) Homepage Journal
      Unlike Charter (who probably uses something not all that different from an mbox file), Google has a global, highly redundant data store that is easier to insert information into than it is to delete from. Even when an email is "deleted" from the GMail interface, there's no guarantee that the data in GoogleFS is actually gone. Google themselves have stated that it may take months (or even years) before the data is purged from the system.

      Which is part of the reason why I actually trust Google with my email. I wouldn't mind them providing a proper backup mechanism (no, POP3 isn't a worthwhile mechanism for me), but it simply isn't as necessary as some hosting providers.

      That being said, this entire mess could have been avoided if someone took a tape backup before purging data from the system... :-/
      • Re:Crap (Score:4, Informative)

        by MightyYar (622222) on Friday January 25, 2008 @12:44PM (#22182720)
        Gmail also offers IMAP now... Just sync up once in a while and you have a complete snapshot of your mailbox - folders and everything.
        • by gallwapa (909389)
          Except Google's IMAP takes EONS to complete a single sync, even after you've already downloaded your entire inbox.
          • Re: (Score:3, Informative)

            by canix (1176421)
            Sounds like you need better software. I use offlineimap and it syncs a couple of family Gmail accounts in seconds.
          • by MightyYar (622222)
            It does seem slow now that I think about it, but I've never really paid much attention.

            Why do you care about time for a backup, though? Not criticizing, just wondering.
        • Re: (Score:3, Interesting)

          by cayenne8 (626475)
          I just try to avoid the problems of letting someone else be in control of my email...so, I run my own email server. If I blow it away or forget to back it up..shame on me, but, I don't have to worry about someone going "oops".
      • Re:Crap (Score:5, Funny)

        by veganboyjosh (896761) on Friday January 25, 2008 @01:06PM (#22183242)
        ...I actually trust Google with my email. I wouldn't mind them providing a proper backup mechanism...

        There's always Google paper [google.com]...
      • Unlike Charter (who probably uses something not all that different from an mbox file), Google has a global, highly redundant data store that is easier to insert information into than it is to delete from. Even when an email is "deleted" from the GMail interface, there's no guarantee that the data in GoogleFS is actually gone. Google themselves have stated that it may take months (or even years) before the data is purged from the system. Which is part of the reason why I actually trust Google with my email.

        • Re:Crap (Score:5, Insightful)

          by Michael O-P (31524) on Friday January 25, 2008 @02:26PM (#22184484) Journal
          You have got to be kidding me. I was going to moderate this as idiocy, but I decided to elaborate with a response instead.

          One might be able to reach a person at Charter, but a helpful person? Not on your life. You speak about behemoth corporations, and Charter embodies the worst of corporate bureaucracy. They are total idiots, the left hand doesn't talk to the right hand, and their prices are unreasonable. And yes, I dumped them as soon as I could so I don't have to deal with them any more. But not once did I deal with a helpful person.

          And deleting 14,000 email accounts just shows the heights of stupidity this company has achieved.
          • Re:Crap (Score:5, Interesting)

            by pnuema (523776) on Friday January 25, 2008 @03:52PM (#22185716)
            My choices for broadband are Charter or AT&T. I despise AT&T's business practices, and yet I will pay them whatever they ask for - I'd sooner do without internet all together than be a Charter customer again.

            A year or two ago, our internet access stopped working. Over the course of three weeks, I spent 12 hours on the phone with Charter support. During those calls I was told: they had deleted my MAC address from the database, that they had no record of me ever being a Charter customer (despite the cable boxes in my living room), that they had no record of me having internet access, and that I was stealing cable.

            Finally, I got fed up, and called AT&T, got DSL and Dish. Once everything was working, I packed up all of Charter's equipment, brought it down to a local office, and told them to shut me off. This was a Saturday.

            On Monday, my Dish stops working. I call up the Dish people; they come out around 10 days later to tell me that Charter had disconnected my satellite and had put a terminator on the line.

            Fuck Charter. Fuck them in the neck with a donkey cock. They are the worst of the worst of companies (believe it or not, this is the short version - the long version is worse). If they actually had to compete, instead of having a monopoly, they would have been out of business years ago - and I would have bought some of their assets so I could set them on fire.

            Fuck Charter.

      • by Otter (3800)
        Unlike Charter (who probably uses something not all that different from an mbox file), Google has a global, highly redundant data store that is easier to insert information into than it is to delete from.

        And yet they also lost a bunch of accounts a couple of years ago, albeit far fewer than in this case.

        • Re: (Score:3, Informative)

          by AKAImBatman (238306)

          And yet they also lost a bunch of accounts a couple of years ago, albeit far fewer than in this case.

          They also restored a lot of the lost data. Something which Charter is completely incapable of.
    • Re: (Score:3, Informative)

      by alexgieg (948359)

      I am one of those people who uses Gmail as a backup betting it's more reliable than my hard drive.

      What I do is this: I use Gmail as my main e-mail address, but I have it set up in IMAP mode at my job's Thunderbird. In my home, I also have Thunderbird downloading e-mails, but in POP3 mode. Why, you ask? Simple: because there I have many special filters set to distribute my mail to special mailboxes so as to make it easy, and fast, to backup them, in encrypted form, to Amazon S3 [amazon.com] using Jungle Disk [jungledisk.com] (together wi

  • Email? (Score:5, Funny)

    by Laguerre (1198383) on Friday January 25, 2008 @12:33PM (#22182542)
    Did they send an email to notify people of the $50 rebate? My inbox is empty...
  • by Capt James McCarthy (860294) on Friday January 25, 2008 @12:37PM (#22182610) Journal
    I've always told users, that email is not a storage medium. It's a volatile one.

    Yes, they should have had backups now days, but none the less, if you want it saved, don't leave it in your inbox.

    I've had folks complain that the trash automatically was cleaned out every three days. WTF?
    • by Otter (3800) on Friday January 25, 2008 @01:19PM (#22183316) Journal
      I'm sorry, but that's ridiculous. Users are absolutely entitled to expect that their provider doesn't delete their mail with no backups available. That's the whole point of using a webmail service, that Google or whoever takes professional-grade responsibility for your data!

      I see no reason why bytes are any more "volatile" in an IMAP file than anywhere else.

      • by garcia (6573)
        I'm sorry but we're talking about an ISP that charges you $39.99/month for service and not your own SMTP server, corporate e-mail account or other pay-for service that ensures that your e-mail is safe. Be glad that they are getting $50 credits. If this was ATTBI or AT&T @ Home, then they would have shrugged and told you "pay for our business class service."
        • by sgtrock (191182)
          Wow. I pay that price for far better service than you seem to expect, yet I expect nothing less. Incremental backups done nightly is NOT that tough an admin task to set up and manage. If a company can't manage that minimal sysadmin task, they have no business pretending that they're an ISP in the first place.
      • by GreggBz (777373) on Friday January 25, 2008 @02:08PM (#22184208) Homepage

        Users are absolutely entitled to expect that their provider doesn't delete their mail with no backups available.

        I'm not exactly sure what you mean by "entitled to expect."

        Gmail is a free (to the user) service. The user might expect it to be reliable, but they are entitled to nothing. If it all goes away, there's no recourse. Also, in light of all the free competing e-mail services these days, most ISP's offer e-mail as a courtesy and will not guarantee it's integrity. It's probabbly carefully worded in every AUP. To practice a business where you are legally responsible for terabytes of customer data is scary can of worms... let Citibank deal with that.

        I work in the ISP business and can tell you that the overhead for e-mail is greater then any other service provided. Every other piece of vital data here is peanuts compared to the size of our customer e-mail storage arrays. We back it up as often as we can. It's an absolutely enormous amount of data that changes every second. We do the best we can but, Hell no, I don't want to be liable for it. I would hope you don't expect me to. Even though I know it's being rsyncd off site twice a day I still advise customers every chance I get. Download your messages and archive the important ones periodically.

        As the old saying goes, the only person you can rely on is yourself.
        • by Otter (3800)
          I'm not exactly sure what you mean by "entitled to expect."

          I mean it in the context of the original comment: third parties engaging in finger pointing should be pointing at the goofballs who wiped out mailboxes without having a proper backup in place -- not claiming that email is some sort of tempfile that users should expect to disappear without warning.

          Legal recourse? IANAL, and I haven't seen the agreement the users accepted, but my guess is that they got $50 more than they're "entitled to" in that sens

          • Re: (Score:3, Insightful)

            by GreggBz (777373)

            goofballs who wiped out mailboxes without having a proper backup in place

            I have some empathy for these goofballs. I'd say that that many accounts was probabbly about 60GB of data. 60GB out of God knows how many terabytes.

            It would be impossible to centralize that much data, except for perhaps the database to verify users. Mbox data would have to be distributed. That would eliminate any single point of failure, but also increase the chance of a small failure. Say you have 60 servers, and 6,000 disks. Say

            • I accept I only run ~50TB of data for my employer. But every byte of that is on servers that do snapshots (ZFS and Pillar Data, but NetApp, lots of EMC iron, BlueArc, whatever would do just as well). Before we do something committing, we take a snapshot. Takes a few seconds, consumes only the delta. Once we've done the committing thing, if things look bad we have the option to roll back (potentially ugly if changes have taken place) or to fish around in the snapshots and use that to stitch things togeth
    • by MBGMorden (803437)
      Also, sometimes email isn't backed up as a matter of policy. We don't want to keep any emails that we don't have to, and having no backup means that if something is gone, it really is gone. We can't go back and pull it from tape.

      Kinda funny though, our in-house lawyer wiped out his entire inbox (which is all stored server-side) about 2 years ago. I almost chuckled when he asked why we didn't keep a backup and I had to remind him that HE suggested that we not do so for legal reasons . . .

      We actually had a
      • by bkr1_2k (237627)
        Unless your doing business in a shady manner, why on earth would you cite "legal reasons" for not backing up your email?
        • by MBGMorden (803437)
          It's not that we're doing anything shady intentionally. It's more like if given a ton of email to sift through, we could very well be liable for something that we didn't even know about or didn't intend. It's just like shredding documents that go in the garbage. Better to get rid of stuff than worry about something somebody might dig up.

          If say, we were involved in a sexual harassment case. A few harmless dirty jokes passed between coworkers generally doesn't cause any harm (talking about willing parties
    • The statement is still valid: your email server is not and cannot be a reliable long term archive. You are foolish if you leave VALUABLE information on it that you don't ever want to lose or have compromised:

      1. Most access is plain text and subject to snooping/hijacking (passwd/userid/content)
      2. Email is the most abused internet protocol (my opinion) by zombots, spammers, and virus/trojan propagators. ISPs do a lot to counter spam and threatening content but sometimes they get hosed. Or your home machine
    • by kevinroyalty (756450) on Friday January 25, 2008 @02:14PM (#22184310) Homepage

      Your statement about people complaining about the trash(deleted email) being emptied brought back a fond memory of mine. I was early in my IT Admin career (was a programmer for over 10 prior to that), I was working for a major insurance company administrating their CC:Mail network. We had issues of people never emptying their "trash" folder and it was taking a long time to do anything for everyone on the system. We sent out a memo to the entire company telling them that in a week we would start emptying the trash folder nightly around midnight before the backups and other maintenance begins. very few people bothered to pay attention to this memo.

      Here is where it get...funny. A high-muckety-muck (eg: pointy haired VP) called the help desk screaming for his trash folder to be restored as it was emptied without his permission and had important files in there. He wanted to see the guilty parties in his office post-haste. I was part of the team, so I had to go to his office. while waiting outside his door for him to let our team in, I grabbed a trashcan and some vertical file folders and paper from his secretary. (can you see where I'm going yet?) We were let in, and he proceeded to rip us up one side and down the other. Our manager brought a copy of the memo, which he promptly threw away while continuing to yell.

      I asked to speak (everyone else was quietly taking the heat) and proceeded to put the trash can on the desk, put the vertical file folders in the trashcan and put paper in each folder. While I was explaining this analogy to him, I asked if every day his office trashcan was empty when he came into the office. He said yes. I basically gave him the analogy that the trash folder in cc:mail was the same as his office trashcan - whatever went in during the day was retrievable, but at midnight, the office cleaning crew would come in and empty his trashcan and we also would be emptying his electronic trash folder the same way. He blinked, and understood what we were doing, pulled the memo out of the trash and reread it. we were dismissed with no apology back to our offices.

      Note that I was a contractor - I couldn't be fired...just sent away back to my firm to go on to the next assignment. I ended up being there for 2.5 years. I still laugh at that situation even today.

  • Unfortunate (Score:4, Insightful)

    by Pojut (1027544) on Friday January 25, 2008 @12:37PM (#22182612) Homepage
    I know these kinds of things aren't supposed to happen, but sometimes they do. The worst part for the company itself is not the backlash they receive...it's the fact that nothing they do and nothing they say will fix it.

    It's one thing if you have angry customers over something you have control over. It's another thing entirely if your customers are angry at you AND there isn't a single solitary thing you can do. That said, I hope that they are more careful in the future...
    • by Neoprofin (871029)
      Don't worry, as a completely unsatisfied former Charter customer in a city where they have a complete monopoly on cable service I can assure you that they have absolutely no interest in fixing things or in how their customers feel. (6 hours on the phone to move service to a new address?)

      There's actually legislation at the state level right now to attempt to loosen their stranglehold of poor service.
    • by jesterzog (189797)

      It's one thing if you have angry customers over something you have control over. It's another thing entirely if your customers are angry at you AND there isn't a single solitary thing you can do. That said, I hope that they are more careful in the future...

      But this is why you're supposed to be careful to avoid screwing things up this seriously. I appreciate that you're trying to take some kind of sympathetic angle here but I don't see the relevance.

      There were a variety of things that a company with 2.6

    • I've never had to back up email for 50 million people, but I've been responsible for a system with 50 thousand people. We didn't backup our email, didn't even come close to having the resources to do so, and it clearly stated in the SLA that we didn't do backups, and if your email got lost, tough shit. Our customers got what they paid for, since the email was free.

      Mostly likely their asses were covered by their service agreement. I am pretty sure that Yahoo's policy for lost email is "tough shit" as well.
  • "No way"? (Score:3, Insightful)

    by KublaiKhan (522918) on Friday January 25, 2008 @12:38PM (#22182622) Homepage Journal
    They didn't make backups beforehand? What kinda incompetent sysadmins do they have over there anyway?
    • by gravis777 (123605)
      I guess this means they are not Sarbanes Oxley compliant. I see lawsuits here
      • Re: (Score:3, Informative)

        by TooMuchToDo (882796)
        Sarbanes Oxley covers the internal emails of a public company. An ISP's customers' emails don't fall into that catergory. Also, under Sars-Ox, you don't need to retain email forever. Even stating you simply keep mail for six months and purge anything older than that is ok.
        • What about public companies who use Charter for email? Yes, if you want it done right, do it yourself. But weren't these paying customers of Charter? I'm guessing it's somewhere in the TOS, but wouldn't storage/backup be part of what one of their customers is paying for?
          • What about public companies who use Charter for email? Yes, if you want it done right, do it yourself. But weren't these paying customers of Charter? I'm guessing it's somewhere in the TOS, but wouldn't storage/backup be part of what one of their customers is paying for?

            No

          • by Blakey Rat (99501)
            I'm guessing it's somewhere in the TOS, but wouldn't storage/backup be part of what one of their customers is paying for?

            This is Charter. Charter customers are just happy they don't come to your house every day and kick you in the balls.
  • by WaHooCrazy7 (1220464) on Friday January 25, 2008 @12:38PM (#22182626)
    Crap! UNDELETE UNDELETE!!!!!
    • by sqldr (838964)
      Or.. "hey guess what, i'm not wearing any underpants!"
      • Or.. "hey guess what, i'm not wearing any underpants!"
        Oh, I don't know about that. I've worked with four DBAs in person, and I wouldn't mind at all hearing that from Lucy. Nor would I be terribly surprised.

        I'll pass on the other three, though. Especially the dead one.
  • /golfclap (Score:5, Funny)

    by UberHoser (868520) on Friday January 25, 2008 @12:40PM (#22182656)
    Charter Manager: You sure that these are the correct accounts to nuke.
    Charter Employee: Yessiree ! 'Click'
    Charter Employee: Oh shit.
    Charter Manager: What ?
    Charter Employee: 'Surfs over to Monster.com' Oh nothing. Nothing at all.
    • Frightening Words (Score:3, Informative)

      by jetpack (22743)
      I don't get upset or worried when I hear a sysadmin shouting and screaming. It's usually the result of some user doing something stupid but limited in scope.

      I was sharing an office with the lab's sysadmin. One day, while I was happily programming away, I heard the quiet utterance from my office-mate: "Oh, shit." Shivers ran down my spine and I started to panic. I knew immediately that all hell was about to break loose.

      Truly a frightening phrase to hear from your sysadmin.

    • Wouldn't it be more like:

      Charter Manager: These are the correct accounts to nuke.
      Charter Employee: Are you sure that these are the correct accounts to nuke?
      Charter Manager: I've told you once, just do it.

      Later -

      Charter Manager: You nuked the wrong accounts, you'd better start looking at monster.com
  • Sure, users should always back up their email. But so should the providers. It takes very little effort to schedule staggered daily/weekly/etc backups. Basically all you need is to have some flunky to carry tapes and disks around.
    • Re:Backups (Score:5, Informative)

      by Amarok.Org (514102) on Friday January 25, 2008 @01:14PM (#22183264)
      For the size of enterprise that Charter is, this is a non-trivial requirement. Having architected, administered, implemented, repaired, and re-engineered backup solutions for many enterprise environments (some in the petabytes-range each), I can tell you that "very little effort" doesn't come anywhere close. I've also worked on the implementation of a mail environment (very much like what Charter has) for a cable modem ISP, so I'm very familiar with the kinds of challenges these environments face.

      The backup architecture required to efficiently and safely protect this kind of environment would cost easily several hundred thousand dollars and several full time employees to manage.

      Before anyone jumps in with "just buy a bunch of cheap IDE hard drives and rsync, tar, etc...", please don't forget that we're talking about a major server farm, probably in several locations, consisting of likely hundreds (if not thousands) of servers and mail stores.

      More than likely, Charter made the business decision that (as other posters have pointed out) email is a volatile storage medium and their internal checks and balances (RAID, etc) were sufficient for protecting against loss. Obviously, they made a mistake and miscalculation. At the end of the day, however, I suspect they'll implement more checks and balances to protect against human error, but I'd be really surprised if IBM/SUN/etc got a big order for a tape library/upgrade. I just can't imagine a company like Charter spending the money (hardware, consumables, people) to back up "Forward this to 10 people in the next 10 minutes and Bill Gates will give you a hand job" messages.

      • It is not Charter's job to determine what is and isn't important for me to keep. They should have prepared and if they didn't (which they clearly have not), then shame on them.

        I think it's 100% reasonable to expect your e-mail (which is paid for as part of your service) be kept safe and not deleted. I do not think that is asking too much -- if your provider includes that service as part of their product offering.

        If Charter had said "hey guys, we'll give you free e-mail but we aren't going to suppor
      • Re:Backups (Score:5, Informative)

        by autophile (640621) on Friday January 25, 2008 @01:42PM (#22183760)

        The backup architecture required to efficiently and safely protect this kind of environment would cost easily several hundred thousand dollars and several full time employees to manage.

        As opposed to, say, the $700,000k they just paid out.

        --Rob

        • Yes, it's probably cheaper to pay out the 700K then to implement a fully redundant, backed-up mail system.

          A new car built by my company leaves somewhere traveling at 60 mph. The rear differential locks up. The car crashes and burns with everyone trapped inside. Now, should we initiate a recall? Take the number of vehicles in the field, A, multiply by the probable rate of failure, B, multiply by the average out-of-court settlement, C. A times B times C equals X. If X is less than the cost of a recall, we don't do one.

      • Re:Backups (Score:5, Informative)

        by Amarok.Org (514102) on Friday January 25, 2008 @01:56PM (#22184020)
        Yes, I'm responding to my own post...

        Just because I can, I did a couple of bar-napkin type calculations to see what it would take to protect this environment.

        I have no idea what Charter's cable modem subscriber base is, so I took some wild ass guesses. According to Charter's website, they have around 5.7 million customers. Assuming that a 12.5% of them are cable modem subscribers, and each of those accounts has an average of two mailboxes, that gives us just under 1.5 million mailboxes to protect. Further assuming an average mailbox size of 50MB (not unreasonable, given the similar environments I've seen), that's somewhere in the neighborhood of 71 terabytes of data - just for email. That's not counting the supporting infrastructure (authentication, transport, etc).

        So to protect 71 terabytes of data, we need somewhere to put that. Tape is most likely. Let's assume LTO3 (probably the most commonly deployed tape technology today in the open systems world), so we've got a raw capacity of 400GB per tape (don't believe the compression specs, I rarely see more than 600GB in the wild). Assuming daily backups kept one week, and weekly backups kept for a month, we'd need about 1780 tapes for the month's rotation. At $40/tape, that's $71,200 in media. Figure 10% per year to replace failed media, and we've got a first year consumable cost of $78,320.

        Now, to get the data onto our ~$80k worth of tapes.

        Let's figure a 12 hour backup window. (We'll assume that this backup infrastructure will be used to protect some other assets in the other 12 hours) To move 71 terabytes of data in 12 hours, we'll need about 28 LTO3 tape drives (I'll spare you the calculations used to get there - but suffice it to say that I included reasonable overhead and observed real-world performance). At $3k a pop (for quality, supportable, maintainable drives), that's $85k in drives. A tape library to contain said drives will be somewhere in the $100k-$150k range depending on options (redundant robotics, etc). The SAN infrastructure required to connect these drives should be in the $30-$40k range.

        So just tape hardware, lets call it $250k.

        Additionally, we need backup servers to handle all this data. No, cheap 1U Intel boxes aren't going to cut it. You're going to need some serious iron to drive 28 LTO3 tape drives at full capacity. Off the top of my head, I'd say you're looking at 2-3 mid-sized Unix servers (IBM System p, etc) loaded down with 4GB fibre adapters. Easily another $150k.

        Right now, we're just under $500k, and we haven't even started talking about software licensing (Tivoli Storage Manager, Veritas Netbackup, ComVault, etc), infrastructure for the systems being backed up (dedicated Ethernet, or depending on volume, dedicated fibre), miscellaneous supporting infrastructure (power, UPS, air conditioning, etc), and so on.

        Once you've got all that, who's going to manage that? Probably a senior backup administrator/architect (90-120k yearly), a mid-range systems administrator (60-90k yearly), and one or two operators (media handling, etc, 30-50k yearly). So that's $250k or so in salaries to manage this beast, figure a benefit load of 60%, and we're at $400k to employ these people.

        Initial hardware investment : $480k
        Yearly consumables : $8k
        Yearly media storage : $60k (no idea - completely made this one up - anyone with knowledge of Iron Mountain, etc, want to comment?)
        Yearly salaries to manage: $400k.

        Completely ignoring data center costs (AC, power, etc) and software, let's call it an up front investment of $1mil, and a yearly ongoing cost of $500k to support.

        14,000 customers at $50 service credit (not real cash) = $700,000

        So as long as they only do this once every two years or so, they're in the black.

        Sorry, but that's how businesses think. (And yes, there's the cost to customer satisfaction, lost customers, etc, but growth will easily outstrip those losses)

        Damn, posting on slashdot feels way too much like real work.

        • Keep it simple, cupcake.

          ... assuming an average mailbox size of 50MB ...

          Sounds reasonable. A 1 TB [newegg.com] drive costs $240 (don't forget to send in for that rebate). That works out to roughly:

          $240/TB / 1000GB/TB / 1000GB/MB * 50MB = 1.2 pennies

          Of course, there are some additional costs involved, but it doesn't cost a heck of a lot per user to back up the email. The more customers, the more cost, but also the more revenue.

  • I use Thunderbird for my charter.net email accounts. I don't understand why more people don't do the same.
    • use gmail's POP collector to harvest mail from Charter to Gmail, then use Thunderbird to read gmail

      email will be backed up on: Charter's servers, for as long as they can manage. Gmails servers, til judgment day, and thunderbird's mbx storage file, until you have a HDD fail or similar bad luck.

      I am often asked to 'set people up on the net' and that is how I always configure their ISP email accounts.. it gives a little bit of extra resilience and also, free virus/spam filtering
  • by eln (21727) on Friday January 25, 2008 @12:41PM (#22182678) Homepage
    Back in the olden days when everyone POP'd their email and disk quotas on the mail server were in the 5-10 MB range, most ISPs didn't bother to back up email for very long because it was expensive and mostly pointless. These days, however, with everyone pushing huge disk quotas and webmail interfaces, the ISPs must be aware that most people will be keeping their email on the server for long periods of time. If this service were free, I might be able to excuse some shoddy backup practices, but in the case of an ISP your mail service is part of the overall service that you're paying for.

    So, either Charter doesn't back up email very well, or their process to "clear out old accounts" involves actually deleting all of the backups of those accounts as well. I already addressed the issue with the former scenario, but if it's the latter, I'd have to say that's a pretty nasty practice too. Any time you clear out old and "unused" data, you have to assume that you're likely to accidentally hit some false positives, which is one of the reasons we have backups in the first place.
    • by bn0p (656911)
      I used to work for a company that did e-mail software for ISPs (although not for Charter) and they all relied on the redundancy in the file servers (RAID-5 or mirroring) to protect against hardware failures. They had a quota of 2 GB per user and they were not going to do backups of up to 2GB each for 5 million users. Charter would need up to ~27 TB to backup 2GB just for the 14000 users whose data was lost. How would they backup the e-mail for 2.6 million customers? Yes, most users do not use their enti
    • by russ1337 (938915)
      E-mail does seem to be a hassle for ISP's - especially the smaller ones. I don't know why they don't 'partner' with Google, Yahoo etc.... Just direct customers to any number of web-based mail services (or google which also offers pop3, imap etc).

      I'm sure Google would welcome the traffic and customers, and the ISP saves a bunch on $50 payouts. (of course the ISP would need to make it clear that the mail is with Google etc, and limit their liability should Google delete your mail account.)
  • Somewhere deep in the bowels of a server room at 2:14 am...

    clickety clickety (SIGH) clickety (beep)

    clickety clickety (beep)

    clickety (beep)

    clickety clickety (beep)

    click- OHHHH SH**! F***!

  • by maillemaker (924053) on Friday January 25, 2008 @12:45PM (#22182738)
    This, once again, highlights the trouble of using "remotely hosted applications" - you are not in control of your data.

    I always POP my email down to my own local computer.

    At least if /I/ lose it it's on /my/ head.
  • It would be better if ISPs were thought of as ASP (Access Service Providers). Unless you have good reason to know that the provide good email or hosting services (that you could stay with even if you switched ASPs) then simply don't use your ASP's "other" services.

    For one thing, they involve a certain sort of lock-in. For another, the ASP never do the jobs as well as dedicated email or hosting companies.

    Get your own domain name, so that you can switch providers (hosting and email) if you need to. M

  • All of us who are programmers have accidentally deleted important information at least once or twice. This kind of error can be forgiven, since we all make mistakes. But there is NO excuse for Charter to not have had a backup of those email accounts before erasing them. Even the most novice programmer, NA or DBA knows that when there is the potential for fiddling with important data, you make sure there is a backup of that data somewhere. I'd be shocked if there were not a class action lawsuit in the wor
    • Unless you knew you were getting a pink slip via email, in which case you might just want to pay them $50 to "lose" your email :P
      • by mark-t (151149)
        Doesn't the employer have to fire an employee in person? Otherwise, couldn't the employee claim they never got the message anyways, and the employer actually be on the hook for paying them at least for part of the day when the employee shows up ready to work?
    • They don't keep backups of email because they don't want them subpoenaed. If all the email is on-line they can do a simple search in response to a court order. If they have a library of tape backups, they have to go through a lot of tapes to make sure the requested data isn't one one. (I speak from experience, I've had to keep obsolete email systems running so that auditors can access email sent many years ago)

      That said, in that environment, they should never be deleting, but instead marking stuff for
  • I look at friends with several years of e-mail sitting on Yahoo, or Hotmail, or Gmail, and always think that they're rather foolish. Aside from the chance that their entire filing system could go poof at any time, or that the company holding it could go bankrupt, those interfaces just aren't intended for archiving and managing large volumes of e-mail.

    And of course, if your Internet connection goes down you're cut off from everything.

    Still, I can't believe that the ISP doesn't have a backup somewhere.
  • which seems a paltry sum for anyone who was less than diligent about backing up their email
    Like Charter? What company doesn't backup their computer systems these days?
  • Host you own (Score:5, Interesting)

    by snarfies (115214) on Friday January 25, 2008 @01:19PM (#22183320) Homepage
    Back in the dot-com days I worked for a local ISP, and established my online identity over the years. The company died, and I lost my long-established email address. Lesson learned - I obtained my own domain name and webhosting, just on shared servers, mind you, and now I have a portable identity that I have control over. Webhost screwing up? I've had it happen a few times now. I just point the domain elsewhere. I have unlimited POP, IMAP, and even webmail. Multiple spam controls that I can fiddle with. And I don't have to worry about Google, Yahoo, etc fiddling with anything either.

    It isn't hard, either. My 63-yo father is now doing the same thing, as he switched ISPs for the first time now that he can get DSL out on the farm, and he isn't the most technical guy.
  • Remember the story about the disgruntled dba who was going to wipe the company's whole database but his "logic bomb" failed? People were saying "Oh, it wouldn't be all that bad, they'd just restore from backup and lose a day or two of work." Yeah, that's assuming the company did things the way they're supposed to. It always seems like companies do things half-assed. This is just another example.
    • by Jim Hall (2985)

      It may always seem that way, but fortunate for the rest of us in IT, it's not always the case. What you don't hear about, of course, are all the times when an organization has deleted critical data by accident, but was able to restore because they did the proper thing and had good backups of the data. That never seems to make headlines, for some reason.

  • Email not backed up (Score:4, Interesting)

    by Maximum Prophet (716608) on Friday January 25, 2008 @01:21PM (#22183358)
    I know many people are saying that Charter should have backed up the email, but I used to work for AOL, and I know they don't back up any of the email, other than having redundant servers in multiple locations. By not sending your email to tape or other media, they can't be hauled into court and forced to give it up. Once it's gone, it's gone.

    That said, it's standard practice when deleting an account to mark the data as deleted, so that it looks like it's gone to the user, but it's actually pending deletion later. Then, when someone complains or pays their bill, you can restore what was "deleted." After a predetermined amount of time, if you don't complain, a cleanup script deletes it permenently.
  • What about those of us who will cancel their Charter service? How does a rebate help us?

    I had already planned to cancel next Monday when my DSL gets hooked up.
  • It had been working for at least 3 years after I quit using them. It had still been in semi-active use up until six months ago. Tried it today and it no longer works. I was surprised it lasted so long.
  • They are the crappiest ISP in the broadband market I have ever had the displeasure of having to deal with. I am a Charter Customer, but I didn't loose mail because I run my own server in violation of the TOS/AUP. This is hands down the number one reason why it should be allowed.
  • Charter's method of operation sudo purgeoldemailaccounts contents of purgeoldemailaccounts: rm -rf /
  • Store on Server? (Score:2, Insightful)

    by piotrr (101798)
    People still leave messages on server? Worse, they rely on it still being there? Man, I must be getting old, I thought we were past this, but apparently web mail has brought back a few of the 'net's child diseases.
  • poop happens (Score:3, Insightful)

    by Deadplant (212273) on Friday January 25, 2008 @01:39PM (#22183708)
    ..should have had backups... should have been more careful... of course.

    But what's done is done and props to them for a bullshit-free apology.
    Most people are prepared to cut you some slack when you screw up as long as you admit your mistake.

    - recognise what it was that you did wrong
    - claim responsibility for your actions
    - apologise
    - state clearly what you learned and what actions you will take to prevent a recurrence

    Or you could take to legal advice / bush administration route
    - flatly refuse to acknowledge that anything bad actually happened
    - talk about how 'the other guys' screw up all the time
    - start an internal investigation and refuse to comment on the issue while it is under investigation
    - eventually admit that 'mistakes were made' but no, you can't think of any specific examples right now and it was all someone else's fault and you there's no way you could have known it would happen.

  • Spokeman talk (Score:3, Insightful)

    by DoofusOfDeath (636671) on Friday January 25, 2008 @01:43PM (#22183796)

    We really are sincerely sorry for having had this happen and do apologize to all those folks who were affected by the error,' Lamont said

    Note the use of the passive voice, which is commonly done to avoid taking responsibility. It seems like even when they're trying to apologize, spin-doctors can't turn off their instinct of avoiding responsibility for mistakes.

  • the customers could always write to the NSA and ask for their backups.
  • i can kinda sympathize with charter (shudder, omg did i actually type that?!). ok, i can feel for their admins.

    for eight months, i worked for a small-town isp with dsl and dialup customers. we had old equipment and no budget for upgrades. we had an autoloader that would occasionally snap tapes, old drive arrays that would fail with no replacement parts on hand ("whuddya mean, we got harddrives right there" "those are ide, i need scsi3"). backupexec would report completed jobs but find no restorable data
  • ...all 14,000 accounts belonged to members of the US government's executive branch.
  • How 'bout instead of giving everyone $50 when something easily preventable--on the order of duh^10, if I remember high school math well enough--they invest in some fucking backup systems for when catastrophes happen?

  • by jhoegl (638955)
    I worked for Charter back in 03 and 04. I noticed a trend before I decided to stop working there. They are self serving morons. They even wanted their tech support to try and upsell someone. "Yeah, um my internet is down" "Okay we can fix that, while you are waiting for the repair guy want to watch HBO for *such and such* extra a month?". I kindly told my bosses at Charter to shove it up their butt. They hired me to fix issues, and I was damn good at it, not to bring in more money. By the way, 3 months

If it's working, the diagnostics say it's fine. If it's not working, the diagnostics say it's fine. - A proposed addition to rules for realtime programming

Working...