Yahoo! Develops Anti-Spam Architecture

prostoalex writes "Yahoo!, the owner of one of the largest e-mail systems in the world, is said to be developing a cryptographic product that will be offered freely to mail servers. 'Domain Keys,' according to the Reuters article, would require the message sender to authenticate in order for message to come across a trusted e-mail network. The idea has been around for ages, however, it required someone from the big league like Yahoo! to step in." While Yahoo! isn't the first name that comes to mind when I think of trusted email, it's still a step in the right direction.

Oh yeah it seems like a good idea right now....

But ultimately one has to worry about the lock that Yahoo! might have on servers once they get it installed all over the place.

Could you imagine this becoming really popular and then Yahoo! getting bought by someone like oh say Microsoft? (or any other big commercial interest)

Re:Oh yeah it seems like a good idea right now....

It can be open sourced, but that doesn't mean anything about preventing lock-in.

Presumably a 'domain key' is some cryptographic element that authenticates that your domain is who it claims to be. To me this sounds an awful lot like SSL where a third party issues the keys, or acts as a clearinghouse for self-issued keys.

Either way, Yahoo could be the man in the middle acting as either issuer or clearinghouse. Think of it this way, OpenSSL is open sourced, but that doesn't keep the SSL issuers from having a lock on that market.

Lock-in isn't necessarily an issue

It can be open sourced, but that doesn't mean anything about preventing lock-in.

Presumably a 'domain key' is some cryptographic element that authenticates that your domain is who it claims to be. To me this sounds an awful lot like SSL where a third party issues the keys, or acts as a clearinghouse for self-issued keys.

Either way, Yahoo could be the man in the middle acting as either issuer or clearinghouse. Think of it this way, OpenSSL is open sourced, but that doesn't keep the SSL issuers from having a lock on that market.

I don't see how lock in will be an issue. Imagine the following scenario:

1. Originating mail software sends a message, including some token in the header that is encrypted using the sending mail server's private key.
2. Zero or more intermediate mail server pass along the message.
3. The destination mail server receives the message.
4. The destination mail server looks up the domain of the message originator and requests that domain's public key.
5. The destination mail server attempts to decrypt the token.
6. If the token is successfully decrypted, the mail is delivered. The receiver knows the identify of the sending system with certainty. Email domains can't be spoofed.
7. Otherwise the message is dropped.

I can't see how this would neccesitate a clearinghouse.

Re:Trusted email?

I use Yahoo mail and its very good.

They have a pretty good spam catching service.
It puts suspected spam in a "Bulk" folder. You can
review this folder or just like it get purged after 30 days. Nice. You can also click on the "its not spam" / "this is spam" buttons to help them tune.

They offer a SSL login and it was discuessed recently on Slashdot that they use the Javascriptcrypto library to calculate MD5's on the client side and send the digiest for seduvcity (maybe when you are not logging in with SSL).

You can check your POP3/IMAP mailboxes. The resources come back color-coded.

Good uptime. Always available.

It's free. You can enought resources for reseaonable use. But you can buy more if you want.

All this sounds exactly like a crypto-nerd and slashdotter would design a mail service. And this new thing is going to be opensourced!

Oh come on!

SpamCon's Barrett cautioned "It's a good approach for those that are willing to use it," he said. "Any kind of cryptographic solution is going to involve some computing overhead, and that's not cheap."

Whereas the latter completely true, I think the weakness of the argument is a testament to the idea being an excellent one. CPU horsepower is very very cheap. If Yahoo think they can do it, then who exactly will have a problem ?

Just as long as I can incorporate it into my server, I'll be a happy bunny - all the other proposals put forward so far seem to limit the mail providers to the big boys ...
Simon.

Temporary

But how am I going to get my special penis enlargement information now? And what about that family matter I am resolving with Mr. Mobotu?

In all seriousness, I think this is a good idea. But, sadly, it's going to be cracked. Domain keys can be forged, and that will be the first thing that these spam servers will be focussing on right now. They'll set up a Yahoo acct and monitor traffic to see what the domain keys look like. They will then duplicate the acks and be back in business. It's only a matter of time.

This is a good step, no doubt. It is just that we should be looking at ways of putting spammers out of business, too. Hit their wallets, not their tech. Tech can always be worked around, especially by dubious people.

Instead of domain keys, I had a different idea that might work a lot better.

What if nobody sent email over the Internet?

Today we have the ability to use web forms to pass messages back and forth to other users on the same service. With that option, the server admin would be able to flag spammers and ban them. If you wanted to message another user of another server, you could type in their location as USERNAME@DOMAIN, and that would queue to be sent in batch to the other server after authentication.

No outside contact. No spam. One message per customer. If you send more than a certain number of messages in a day, they are held as possible spam.

Privacy goes out the window, but hey... it's not like there is any privacy in non-encrypted email anyway.

Not necessarily

If they use decent encryption, cracking this scheme will be nearly impossible. If they use a digital signature algorithm such as DSA or MD5, or public key algorithms such as RSA, the computational power required to crack these keys will be far beyond the means of the richest spammers.

Personally, I'd like to see two things.

1. The software Yahoo! is developing should be open-source, so nobody can monopolize it. At the very minimum, the protocols involved should be well documented so open-sourcers can make their own implementations if they have to.

2. Give this software a few months to propogate to a good chunk of the ISPs out there. Then, Yahoo! should announce that they will NOT accept any email that is not signed with this software. I'll guarantee that everyone will be using this new protocol in a matter of weeks, since no ISP wants customers screaming because they can't get mail through to Yahoo! accounts.

Must be missing something

The text of the article has to be wrong - they say the private key is delivered as a message header! Hmm, not very private...

I'm assuming that what is sent out is an encypted token for which the public key can be used to decrpyt, so:

• Alice wants to send an email to Bob.
  
• Alice encrypts the MD5 checksum of the mail body content (or some other representative text, probably longer than 32 bytes!) using her private key, and embeds the resulting encoded string into a mail header
  
• Bob receives the mail, and looks up Alice's public key to decrypt the token
  
• Bob compares the decrypted token with the same representative text to see if they match.
  
• Match => Read. No match => Put into 'Junk' folder
  

So, the token to be encoded will change from mail to mail, thus making replay techniques pretty much impossible, I think. At least, that's the way I'd do it, and I'm pretty sure I've seen it presented before as well...

On the other hand, I ain't a security expert, so there's probably a gaping hole in the above :-)

Simon

OS?

Does anyone know what software Yahoo's mailservers run?

Re:OS?

$ telnet mx1.mail.yahoo.com 25
Trying 64.157.4.78...
Connected to mx1.mail.yahoo.com.
Escape character is '^]'.
220 YSmtp mta108.mail.sc5.yahoo.com ESMTP service ready


It looks like they run YSmtp, just like everyone else I know. In all seriousness, I'd imagine there isn't much of Yahoo's infrastructure that isn't highly optimized for Yahoo's own use. I think that Yahoo did a lot with FreeBSD at one time, but I'd presume whatever they have isn't just an out of the box app.

So now...

...you'll only be spammed by Yahoo??

Re:So now...

yes, but now you'll know for sure that the email came from Yahoo - and not some forged return-to that dumps on some ordinary Joe's server.

step, by step, the spam problem can be solved. That doesn't mean that you should not take the first step simply because it doesn't provide a total cure.

Open standards?

As long as it's an open standard that eventually becomes RFC3821, I'll be okay with it. But if it's one of those proprietary "pay us to participate" schemes, they can go jump. Oh, and there should be no scope for someone to say "pay us or we won't accept email from you.

You just can't win with the /. crowd

If someone announced a cure for all cancers, this crowd would immediately dismiss it because it could possible be bought by Microsoft. You pimply-faced pessimists remind my of Eor from Winnie the Pooh.

Re:You just can't win with the /. crowd

You pimply-faced pessimists remind my of Eor from Winnie the Pooh.

No, Xor is the operation most often used in cryptographic functions...

Broken already?

One guy's take on why it won't work [megacity.org]

Not sure if I understand it right

How do they propose to keep the encrypted private key secure? I did RTFA but couldn't find any explanation of how the encrypted version of the private key could not be spoofed since it is part of the message header.

If the spammer...or anyone for that matter is spoofing a header anyway, it shouldn't be difficult to find out the encrypted private key, since it is sent out with every message originating from the domain.

I could, presumably send an email from my secure email address to a non-existent email address of the domain whose encrypted private key I wish to find out: eg [e-mail]. The bounced message should have it in the header.

Re:Not sure if I understand it right

How do they propose to keep the encrypted private key secure? I did RTFA but couldn't find any explanation of how the encrypted version of the private key could not be spoofed since it is part of the message header.

If the spammer...or anyone for that matter is spoofing a header anyway, it shouldn't be difficult to find out the encrypted private key, since it is sent out with every message originating from the domain.

I could, presumably send an email from my secure email address to a non-existent email address of the domain whose encrypted private key I wish to find out: eg [e-mail]. The bounced message should have it in the header.

The authentication token would likely be some sort of hash of the message contents. In that way, a token is only valid for that particular message. The sender would generate a checksum of the message, encrypt it with a private key, then transmit the encrypted checksum as the token. The receiver would generate the same hash of the message contents, and decrypt the token with the public key. If the decrypted checksum equals the generated checksum, then one can be confident that the message came from the server it said it came from.

So what about a teergrube?

The first time that I heard about a teergrube [iks-jena.de] to use as a way to block -- or at least make it damned difficult for -- spammers I was intrigued at its simplicity. And tho' I find references to it all over the 'net, I don't think that it has been mainstreamed yet, and frankly I don't know why. Have spammers developed a counter to a teergrube? Or do mail admins simply not know enough about them?

One solution

when you think about it, BUT this should come from IETF or some other body not from a company. A few important points:

1) Who will issue the keys?

2) Is anonymous mail possible if the receiver allows it?

Furthermore spamming is a social problem emerging from our commercial world and technical solutions can never be 100%. What if:

a) I send spam from a "secure" domain?

b) forge certificates?

c) the certificates are too expensive? (like SSL, I think it should be included with a domain)

I like the "Bayes" spam filters best. You get 99.5% spam protection and keep anonymous mail.

We all see the need for authenticated senders (biz communication, etc.), but we should be careful ...

Re:One solution

... this should come from IETF or some other body not from a company.

We should expect something like this to come from the IETF, but big corps do good things all the time. What makes you uncomfortable about it? The privacy issue? If it's on the net and you want privacy, encrypt the content. But if you want to hit my network w/ SMTP, much less an ICMP package, I want to at least know who you are.

Are you worrying who will govern the entire thing? Who do you trust? Some .org run by someone? Some corp? The gov't? All-in-all, you have to trust SOMEONE.

romancing the stone

AOL has recently started banning SMTP servers who don't have reverse addresses, as seen on the NANOG lists. Personally there are so many methods to eliminate spam that an administrator can take I don't see what the issue is.

Me personally, if spam makes it through my filter, I ban off the offending address working my way up towards the class c - b - a. All attempts at a port 25 connection is drop point blank, http, https, etal are kept open. I also have dontspam#somefreemailaccount.com's to use for form shit. Once in a while when registering for say an upper-crust website account, I'll use something like [e-mail] to see who exactly is sharing my addresses, then null the account if I see anything odd coming in to that account, and never trust the site again. Procmail works the most wonders though.

User account verification

First let them implement some user account verification, so that a RCPT TO: results in a 550 reply when that user does not exist.
This enables SMTP callbacks to stop spam being spoofed "from yahoo", just like everyone else does.

So where's the info?

Okay, so they're developing a system that they'll release to open-source developers.... why not DEVELOP it in the open in the first place?

Are cycles that cheap?

As I understand it, the proposal requires public-key encryption for every email sent, done by the sender at the time of sending. (If the "private key" -- something encrypted with the private key -- could be computed once and reused in every message, it could be copied and replayed by a forger.) This can dramatically raise the overhead associated with sending mail. Perhaps that overhead is reasonable, perhaps not.

Bala Krishnamurthy at AT&T Labs has given a number of talks recently, including to the IETF, on a spam disincentive program he calls SHRED [att.com]. My understanding is that it uses offline cryptographic computation to amortize this overhead and distribute it to parties willing and able to devote the computational resources.

In any case, the tag line for this article had it right, standardizing this will be hard and heavy-hitters like Yahoo will need to take the lead. But a key problem is getting the new system to interoperate with the old.

Only for GPL players?

From the article: Yahoo said its "Domain Keys" software, which it hopes to launch in 2004, will be made available freely to the developers of the Web's major open-source e-mail software and systems.

But later: Garlinghouse also argued that Yahoo's proposal should be attractive to other e-mail providers because it is free and comes with no special restrictions. Is the GPL considered a "special restriction"? Will it not actually be GPL, just available to open systems?

I'm guessing that you'll need to be a GPL mail server to both require the private key for receipt, and to be able to use the system to give the email the private key for sending. So, what will this do to non-open mail systems?

• You could presumably send to a non-open system, as they will simply ignore the key if present, but will still accept email if absent.
• Open systems that require the key to receive will presumably refuse email without the key (otherwise what's the point), which means that a mail system that's open that uses this methodology might gain the perception of "being broken" from the end users point of view. Of course, the admin setting up such a system would be well aware that some email will be refused, and will be prepared to handle refusals, either with a "bounce message", a phase in period that just gives a warning, etc.
• Senders that use a non-open system that can't use this technology will find an increasing amount of their email being refused; at first they'll blame the recipient, but as this gets more widespread, they'll blame their own sending service. Is that the sound of IIS's mail server being obsoleted?
  
• The end result will be that users of open systems will receive less spam, whereas users of closed systems will find themselves still receiving spam, and increasingly unable to send to others.

Is Yahoo trying to break MicroSoft's mail service? Will this work? What's MSFT's option--reverse this and include it in their system anyways? Switch to an open system for a mail server, like, say, something based on a BSD license? Or ignore it, in an attempt to deprive it of critical mass?

Indeed, this might all be moot; Yahoo might make it free and available to everyone, either on a free system or a non-free system; the article isn't clear as it says both. It could also be that MSFT already uses an OSS mailserver in IIS for all I know about MSFT product. But I suspect this is a power-grab, like everything else these days. And, I have to say, if it is I wish Yahoo the best of luck--this would be another demonstration of the power of OSS; it allows the community to change together on a dime and play well together. Whereas makers of proprietary systems each have to modify their own systems with their own coders.

identity based antispam is censorship tool

a thing to remember is that if someone can prevent a spammer from communicating based on identity (or lack thereof), you can be silenced as well.

This is why I have put my efforts into sender-pay systems and specifically the camram project. We invite you to please come and join us in the effort to build a decentralized, user-friendly, freedom-of-speech supporting antispam system and hit spammers in the pocketbook.

camram antique documentation [camram.org] (too busy writing code to write new documentation)

BEWARE THE BIG RED Y!

If they're offering it for free, BEWARE. IT'S A TRICK. There's some hidden patent they're going to decide to enforce once the entire world adopts the architecture.
*waves hands ominously*

Yahoo beats eariler proposals? I hope not.

Would you rather choose a Yahoo product over an open standard that is under development? I'm speaking of AMTP [ietf.org], of course. (See AMTP author's site). [bw.org]

Yahoo's size doesn't give that much weight to their proposal. Yahoo's email is not used in business to business communication (do not count hot dog stands as businesses), so businesses can just aswell block everything that originates from *@yahoo.com if it is not directed to their consumer service department.

Also, reverse mx [mikerubel.org] records provide much of the same benefits with minimal alterations needed to current email infrastructure. One DNS record added and small change in MTA software.

If Yahoo would really like to do a service to the internet community, they should rather consider looking AMTP and reverse mx records.

Too resource intensive, and broken anyway

Under Yahoo's new architecture, a system sending an e-mail message would embed a secure, private key in a message header. The receiving system would check the Internet's Domain Name System for the public key registered to the sending domain.

If the public key is able to decrypt the private key embedded in the message, then the e-mail is considered authentic and can be delivered. If not, then the message is assumed not to be an authentic one from the sender and is blocked.

For every message, I have to check and unpack the header, go out to some PK server, and validate the keys, before I decide to accept/reject? That introduces a big latency into SMTP.

Also, this doesn't do anything to stop 'legitimate email marketers'. There's a death penalty (blacklist) for a site or particular sender's key, but nothing to stop a spammer from changing keys and starting over.

Or will everyone have to get their own key pair? Who's going to validate them, and at what cost per key pair?

This won't do a thing to stop spam, and imposes too big a burden on the infrastructure and on the 99% of us who don't spam.

Not for me

Require the message sender to authenticate in order for message to come across a trusted e-mail network

Read: trusted network == commercial network

Why do you think this is in the "Money & Investing" department (see the linked article). No, this isn't for me. Businesses may well choose to use something like this for their communications, but they will not have the pleasure of communicating with me. While SMTP has its flaws, it still allows any IP host to send mail to any other IP host and that is a good thing.

To gain insight into what's going to happen with email and Internet communications in general over the next couple of years, you have to adopt a business mindset to see it from their eyes. There is a big problem (spam) hence a potential to make money. Various companies are going to try and cash in on this situation by offering a solution that might very well decrease spam -- some sort of commercially controlled communication network -- but this is definitely not in the best interest of the Internet. Of course, it's in the best interest of the company that's peddling the solution (duh!)

The Internet isn't Compuserve, or AOL. It's a network of IP hosts, and those are the entities which should have a facility for sending communications back and forth. There is no need for a central carrier for communications

Leading to a standard

The way the IETF and other standards bodies have worked is that some organization wouldtry out a new concept for a technology and once they feel the concept is working, they will create a Request For Comments (RFC) which allows others to implement and offer feedback. Over time the RFC gains support and ultimately becomes a recommendation.

This process was used to create the internet today, including all of the network protocols and services that run on top of it. Even SMTP was an RFC first.

This is a large, stinking pile of bullshit.

1. This is the classic confusion of authentication with security. Authentication does not protect against spammers. The spammers will simply authenticate and keep right on spamming, and now they won't have to do tricks to circumvent the filters because the cert makes them "trusted". (One other example of this is the illusion of security caused by cryptographic authentication on the web. That hasn't stopped spyware sleazebags such as Gator/Claria; they just get their own certs.)
2. Yahoo is an unrepentant spammer [google.com] and spam support service [google.com] itself. They reset your marketing preferences [slashdot.org] at their whim. Abuse reports routinely go to /dev/null. Any "anti-spam" solution coming from a spammer and spam supporter is necessarily a scam.

Public key spam control - technical implications

This looks like a variation on the scheme to use DNS to distribute public keys for encrypted mail. It could even use the same key.

The basic idea, as I understand it, is that the DNS for a domain holds a public key, and mail sent with a "from" address in that domain must be signed with that public key. That's an old idea, and not all that bad. You create your own public/private key pair; you don't have to buy a "certificate" from somebody. (I think.) If you control a domain's DNS info, you can send mail from anywhere with that domain listed as the sender, as long as you know the private key.

For the free-mail services, it's fine. All their mail is authored via web applications and sent from their own servers. Only the service has the private key. Only the outgoing SMTP servers need to know the private key. That's the Yahoo Mail case.

If you own a domain, you should have full control over your own public and private keys. But adding additional info to a DNS record is not well supported by most hosting services. If you're not running DNS yourself, you may have problems setting your public key. Hosting services have to support this.

Signing can occur either in the original user agent (the SMTP sender) or in a mail forwarder. It's easier to implement this in mail forwarders, but if you want to send using a return address other than the one of the mail forwarder you're using, your user agent has to know how to sign mail.

If you're downstream from an ISP and don't control a domain, the ISP owns the key for the domain and can control what they sign. That has implications. They might force you to use web mail, for example. Or run their client software on your machine.

Spammers can still register domains, run their own DNS, sign their mail, and spam. It doesn't really stop spam.

Your public key is now valuable, and a target for spyware and viruses. Expect to see viruses that steal public keys from (inevitably) Outlook and send them to spammers. Or just send spam from the attacked machine.

What this really does is provide a clear way to identify joe-jobs using addresses from major mail services like Yahoo Mail. That helps Yahoo more than anybody else.