EarthLink Is Losing a Lot of Email

LandGator writes "Robert X. Cringely, doyen compu-columnist for PBS, reports on a hidden e-mail problem at Earthlink: They're losing up to 9 messages out of 10, found as a result of a friend's testing." From the article: "He sent messages from other accounts to his Earthlink address, to his aliased Blackberry address, and to his Gmail account. For every 10 messages sent, 1-2 arrived in his Earthlink mailbox, 1-2 (not necessarily the SAME 1-2) on his Blackberry, and all 10 arrived with Gmail. Swimming upstream through Earthlink customer support, my buddy finally found a technical contact who freely acknowledged the problem. Since June, he was told, Earthlink's mail system has been so overloaded that some users have been missing up to 90 percent of their incoming e-mail. It isn't bounced back to senders; it just disappears. And Earthlink hasn't mentioned the problem to these affected customers unless they complain."

They DID notify customers

However, it was via email.

"Nothing for you to see here..."

Is earthlink hosting slashdot?!

Lucky for Earthlink Customers!

Less spam. Enjoy.

DIY

This sort of thing is the reason I host my own e-mail. At least this way I usually know when it's broken, and I have the opportunity to fix it.

Re:DIY

This sort of thing is the reason I host my own e-mail. At least this way I usually know when it's broken, and I have the opportunity to fix it.

Technically that is against the ToS for regular Earthlink accounts.

Secondly they like to block a lot of traffic on email-esque ports.

Either way... As a former employee, I'm not surprised.

Re:DIY

Essentially true.

Though I wouldn't use the word "inept".

Try putting a couple hundred domains and 10k users on it and your threat surface for spammers goes up exponentially from a small server with a few domains and a couple hundred users.

Ours gets tens of thousands of bogus connection attempts from spammers per hour. How many are you getting? 50? That's not including the stuff that does get into the filters to be processed by the rules.

Until you have run a big box with lots of users on it, you have no freaking idea what we deal with on a daily basis. And it has gotten MUCH worse in the past 18 months.

Lost e-mail? WHAT THE HECK?

E-mail should never be lost! We had an issue where I work with e-mail BACKING UP for a few months while we implimented new mail servers... but no mail was ever lost.. it either got bounced back (not usually) or would arrive several hours after it was sent. To actually LOOSE e-mail indicates that Earthlink is ACCEPTING the mail and then DUMPING it!!!! When our servers were overloaded, we just rejected the connection, until the mail server could handle more mail.. and then we accepted it.

Re:Lost e-mail? WHAT THE HECK?

"Loose" is clearly spelled with two Os. "Lose," on the other hand, is spelled with one.

Re:Lost e-mail? WHAT THE HECK?

One might note that, but one would be wrong. The "no apostrophes for plurals" rule doesn't go on vacation just because it's a single letter.

There are style guides that disagree on this point, granted, but the ones that disagree with me are self-evidently wrong.

Re:Lost e-mail? WHAT THE HECK?

E-mail should never be lost! ... To actually LOOSE e-mail indicates that Earthlink is ACCEPTING the mail and then DUMPING it!!!!

This is absolutely correct, so any policy checks that occur during the SMTP handshake (who are you? where are you coming from? who do you want to send to? how much data do you have? Oh, do you now? REJECT). However, anti-spam and anti-virus checks happen after the message is accepted. If the result of the check is X, and policy rules say drop mail on the floor when X, then bye-bye e-mail and sorry Bond, the government will not ackowledge its involvement.
Otherwise, the only way to loose mail is to shutdown a machine with a heavy queue and throw out the disk. SMTP is impervious to network badness. My money is on an SMTP policy run amok.

Re:Lost e-mail? WHAT THE HECK?

However, anti-spam and anti-virus checks happen after the message is accepted. If the result of the check is X, and policy rules say drop mail on the floor when X, then

Then it's a horrible policy.

Every single email service I've signed up for that does spam filtering has a "spam" or "bulk mail" or "junk mail" folder. I implemented my own when I deployed my own personal server.

And virus scans should be able to remove the virus and tag the message as "WHOOPS THIS HAS A VIRUS", but shouuld not drop them on the floor.

If this is causing undue stress, you could implement policies in the handshake. "Oh, I've gotten 100 emails from you in the past hour, and 90 of them were spam/viruses. REJECT." Or put it in some sort of tarpit.

An email service losing email is somewhat like anyone losing data these days. There is absolutely no situation where email should completely disappear into an email server and never come out.

SMTP Digression

I think you have an unrealistic expectation of the quality of service you can expect from email.

I wrote an SMTP server a while back (to check out Ruby's network libraries), and while going through the RFCs I found that there are expectations of quality that include delivery of something. It all comes down to the protocol: if a server accepts a message, it takes responsibility for the message's delivery. A server should reject the message if it cannot deliver (causing the delivery server to either try again later or tell the user there's a problem).

To be losing mail, Earthlink servers must be accepting mail and then throwing it away, or at the very least, not continuing to forward it to the destination, which is just as bad. This goes completely against how the system is supposed to work. If they can't handle the load, there's a specific set of return codes to give (RFC821 [faqs.org], section 4.2):

450 Requested mail action not taken: mailbox unavailable [E.g., mailbox busy]
550 Requested action not taken: mailbox unavailable [E.g., mailbox not found, no access]
451 Requested action aborted: error in processing
452 Requested action not taken: insufficient system storage
552 Requested mail action aborted: exceeded storage allocation
553 Requested action not taken: mailbox name not allowed [E.g., mailbox syntax incorrect]
554 Transaction failed

I understand your perspective -- email is a loosely connected system, with lots of points of failure. However, in the vast majority of cases, a failure at one point will cause either delays or errors, not dropped mail.

Re:Lost e-mail? WHAT THE HECK?

Actually we handle close to 50,000 e-mails per minute.... I happen to work for a largeish ISP in Pennsylvania :)

What MTA do they use?

I checked their mailservers (what the MX record reports anyway), and they have a very generic ESMTP banner, not really apparent which MTA they use. I want to know which MTA can lose mail because of overload. So I can avoid it like the plague. I do know for a fact that Sendmail and Postfix send a 4xx error if mail cannot be spooled for delivery (for whatever reason), allowing the sending MX to retry at a later time. There is absolutely NO excuse for a mailserver dropping mail like that.

Re:What MTA do they use?

Earthlink uses an SMTP server they developed in house because they felt the existing servers out there would not scale to the level they needed and were not secure enough.

I used to work there back in the day.

Wonder if they can be sued

This isn't a free webmail account, this is something customers pay for. Some people could have lost a lot of business. And what if someone has been searching for a job for the last 6 months and their monster.com etc contact info has only this email address?

Re:Wonder if they can be sued

From their policies section [earthlink.net]

3. THE SERVICE Depending on the type of Service that you choose, the Service may include internet access, software, hardware, email, webspace...

...and further down...

THE SERVICES ARE PROVIDED ON AN "AS IS" AND "AS AVAILABLE" BASIS. EARTHLINK DOES NOT WARRANT THAT THE SERVICES WILL BE UNINTERRUPTED, ERROR-FREE OR FREE OF VIRUSES, OR OTHER HARMFUL COMPONENTS. So while it sounds like its horrible customer service, it doesn't sound like they're in any danger of being sued.

She loves me not

My luck, the one email this chick got from me, was the one with me telling her off for not answering the other 9.

Is this 1998?

1) I didn't know EarthLink was still around.

2) I didn't know Cringely was still around.

If it hadn't been for the reference to GMail, I'd be wondering if this story had been sitting in the queue since 1998. Now, off to buy some LNUX shares, and one of those Tommy Hilfiger straps to hang my keys around my neck!

White list spam block with challenge

I run an online retail business, and non-tech savy customers using earthlink don't get a lot of our email.

Biggest problem is that Earthlink uses a white-list spam blocking setup that sends back a time-limited challenge to the sender ("Please go to this link and fill in this form so that this user can receive your mail").

We get these challenges when our automated system sends messages to customers ... but there's simply no practical way to respond to them all within the time limit (during business hours it's not practical - it's impossible at 2am while you're sleeping). And unfortunately it seems that a lot of their members simply don't understand what it means when you tell them they have to add you to their list or they won't get your email.

- Roach

When asked about the email problems...

... an Earthlink representative said that he had received no notice of any email problems. Next question please.

I bet I know whats happening...

I used to work for a hosting company (a pretty shitey one at that). They refused to update their mail systems so all customers had email issues. They used an old customized version of SendMail on the Linux box's. The problem, the queues were getting full and either locking the mail up or raising the machine's processes up so it could no longer do anything. The solution, delete the mail off the server and kill all processes. This was a temporary solution that turned into a permanent one because "It worked." The problem was, the mail was gone when it happened so all the tech support guys had to hear the poo poo mad customers.

I would wager thats whats going on here and they don't want to admit it. There is some admin there (or it could be company policy) that see's alot of mail getting queued up but not being delivered but instead of fixing the problem he just deletes the mail and everything's fine.

They must have hired the BOFH

and he just piped the e-mail to /dev/null. All that e-mail was taking up too much disk space, anyways.

Not only losing incoming email

Last month or so, yahoo started bouncing email from earthlink. 100% of the time. Calls to support eventually indicated it was a known problem (didn't admit it until pressed), and then indicated multi-day wait for it to be fixed. It was easier to fix on my wife's side; reroute her mail through my hosting server. Though the advice from our 13 year old son was probably the best: "why aren't you using gmail?"

Here's the root of the problem

Have you seen the commercials? They've off-shored half their jobs to magic-fairytale land. They've probably got some under trained ogre for an email admin who stands around the water cooler all day chatting with the fairies. Sure, it's a lot cheaper when you can pay your staff in pixie dust, but you end up getting bitten by poor customer service (I had to call them the other day, and the customer service rep had such a thick elven accent I could barely understand him). Outsourcing just doesn't work...

No guaranteed email delivery

Lets remember way back when this happened all the time shall we.....and we used to say to the users.....

"there is no guarantee of email delivery" (and optionally "Get over it")

Remember this folks, no where in the RFC's is there anything that states email will get delivered....

Just because all us sys-admins do such a great job, most of the time it does get there, people forget the dark ages of the internet when this would happen all the time.

OK 90% email loss is really really bad, and it use to be more like 5% loss (at worst), but people need to remember email isn't guaranteed.
 

Re:No guaranteed email delivery

Just because all us sys-admins do such a great job, most of the time it does get there, people forget the dark ages of the internet when this would happen all the time.

Back then, you may have had an excuse. Today, the excuse that the RFC doesn't specify email gets delivered should get you fired for being a failure who doesn't give a shit. Just my $0.02.

Re:No guaranteed email delivery

Remember this folks, no where in the RFC's is there anything that states email will get delivered....

This is what I thought when I read the headline (having used that excuse in the past ;). So I immediately looked up RFC2821 - Simple Mail Transfer Protocol

6.1 Reliable Delivery and Replies by Email

      When the receiver-SMTP accepts a piece of mail (by sending a "250 OK"
      message in response to DATA), it is accepting responsibility for
      delivering or relaying the message. It must take this responsibility
      seriously. It MUST NOT lose the message for frivolous reasons, such
      as because the host later crashes or because of a predictable
      resource shortage.

      If there is a delivery failure after acceptance of a message, the
      receiver-SMTP MUST formulate and mail a notification message.

naturally there is alot more, including cases where it is acceptable not to send a notification, but I don't think any apply here.

So basically, SMTP is defined as a reliable protocol which guarantees delivery or notification of failure. The days of unreliable e-mail no longer apply.

As An Earthlink Customer

I have no problems. Plenty of spam and the good stuff comes through for me and my wife.

Believe it or not, they've been very good. The one issue I've had they resolved quickly once I was escalated above first-level tech support.

Maybe it's a location-specific issue?

I can qualify another post that talked about the new email sender verification thing. I get it sending mail from the web email interface. But none of my friends or the emails I send myself from work require sender verification.

I don't know what the motivation behind these complaints may be. It certainly isn't bothering me or my wife. Maybe it will..

(shrugs)

Re:Says a lot..

Generally when you're stating statistics like "up to 9 out of 10", that means that in at least one your test runs, whatever it was you were testing hit 9 out of 10 times, in this case the fact that Earthlink lost e-mails. You say "up to 9 out 10," because in other runs it may have been 1 out of 10, 3 out of 10, or 7 out of 10. You're trying to show that how bad it gets. When stating the statistic for useful statistical purposes, however, one should definitely also give averages, like: "overall Earthlink lost 50%". Now, in this case, Cringely actually states that his friend tried several times to send messages a block of 10 messages to Earthlink account, to an aliased Blackberry account, and to his Gmail account. Each time, Only 1-2 made it to the Earthlink and to the Blackberry (aliased off the Earthlink account), and all 10 made it to Gmail.

Re:Says a lot..

Parent is +4 insightful? Wtf? The quote specifically says the average was 8-9 lost. Parent is implying that maybe only 1 was lost for the most part, or something similar.

C'mon, can people not even be bothered to read the article SUMMARY any more?

Re:Says a lot..

I turn green up to 9 out of 10 times I take a shower

Can I watch?

Re:Says a lot..

Usually the phrase "up to" is used to make the data sound like it says more than it does. It literally means "possibly as much as but no more than", but people read it as "on average". Using "up to" to make a claim that sounds broader than it really is is, as you imply, dishonest.

In this case what they have is an estimate of the average. But it's not based on much data or systematic testing. By saying "up to" they are actually encouraging the reader to interpret their results narrowlly. They could have said "9 out of 10 times", but by saying "up to 9 out of ten", they imply (a) they have no evidence that the system performs worse than this and (b) the system may at times perform better than this.

Overall, I think this is an example of honesty.