O'Reilly Article on Spam Defense

Dru writes "Here's an article talking about the details of implementing a network level spam defense with Qmail. It also talks a little about a new site called Trustic which uses a trust system (like Advogato) for nominating spammer/hostile IP's."

hostile IP's

for nominating spammer/hostile IP's

and thereafter all packets from said IP's are market with the Evil Bit.

Re:hostile IP's

I propose that, rather than changing content, proxies simply add the evil bit to packets from sources that they know to be evil. This can be treated by applications as simply a suggestion, like CSS. Here is how we can set the evil bit---at the proxy level! Mark banner ad transmissions as evil!

Wow!

I never realized that ole Bill was such a tech expert!

I love qmail.

I suggest buying the book [amazon.com] if you plan on implementing it. The online version [lifewithqmail.org] isn't enough (and covers about 1/3 what the printed version does).

Make sure you follow the relay-ctrl section very close. You could be a source of spam if you do it wrong!

Sounds neat, but PGP'ed network sounds better.

I though of this when it comes to SPAM:

Have a computer certified by another individual and create a public/private key for that computer. Do this step to create a network of ID's for the servers.

Now, have admins "Sign" a certain public text that allows servers to trust other servers.

If Company X is being real lax (eg: promoting spam), write a revoke key and put it on a few OTHER machines. Thien it'll propigate throught the mail-net to disallow all connections from that MAIL server.

Of course, mail servers and clients would have to have different trust relationships ala ssh.

For them mail geeks: would this be feasible? I could see CPU load go rocket...

Re:Sounds neat, but PGP'ed network sounds better.

If Company X is being real lax (eg: promoting spam), write a revoke key and put it on a few OTHER machines. Thien it'll propigate throught the mail-net to disallow all connections from that MAIL server.

Just curious, how is this different from a blacklist? It sounds like the same concept, just different technology.

Re:Sounds neat, but PGP'ed network sounds better.

Having to generate and spread keys and key revocations non-stop sounds like a very high maintenance system.

Well, at least that would give some techies back their jobs, although I'm not too sure they would like their new job...

Regards,
--
*Art

Trusted IPs

>for nominating spammer/hostile IP's

Also for nominating trusted IPs.

Hurrah for blacklists

now all we need to ask is how long till this "community" service that they provide will take before they start charging $ for querying it just like every other blacklist, making blocking spam a privilidge for the rich (i believe MAPS is over a 1000$ a year)

Re:Hurrah for blacklists

You're probably right, they will eventually want to charge money, and, IMHO, thier solution looks overly complicated and manipulable (spammers pay for "trusted" members to list them as "trusted").

It would be better if ISPs participated in services like the ORDB [sordb.org], SORBS [sorbs.net] and Monkeys [monkeys.com] that have simple network testable criteria for listing open relays. Spews [spews.org], Spamhaus [spamhaus.org], and DSBL [dsbl.org] have reputable lists of usernames and addresses that send spam. If ISPs and admins would participate in projects like these, the spam problem would be greatly reduced. And it seems that these projects are mostly run by admins who are interested in blocking spam, not selling a service.

By the way, MAPS [mail-abuse.org] is currently free for individual use [mail-abuse.org] (look at the bottom of the page).

Distrustful of Network Level Censorship

No thanks.

Your spam may be my correspondence -- I may want to get mail from those whose conduct you find abhorrent. Today, a network may responsibly be censoring only unwanted and unsolicited commercial e-mail. Next week, the powers-that-be-in-the-networks start censoring geek news.

To protect our liberties, spam control should be decentralized -- as close to the last mile as possible. Yes, of course, this means that the supposed great harm of spam -- huge volume transmissions through the network -- will not be interdicted closer to the source. In my view, an effective end-point spam model is as likely to reduce volume as a network centered model: the idea is to reduce the INCENTIVE to spam -- that will reduce the volume.

Centralized technical measures simply invite the spam wars to continue, provide centralized points of failure, will not diminish spam, and will assure that powers-that-be have ample new abilities to censor speech.

Re:Distrustful of Network Level Censorship

To protect our liberties, spam control should be decentralized -- as close to the last mile as possible.

It is. I'm the one deciding whether or not to use this service.

Re:Distrustful of Network Level Censorship

problem is too many of you are deciding TO use it. AOL, Hotmail, MSN to name a few.. the 'want' to filter spam at the server level hurts legit email marketers, inconveniences recipients of legit email marketers, and to the parent's point - creates a target for spammers.

server side email filtering is BAD, BAD, BAD!

what if the US Post Office started throwing out your clearing house sweepstakes and credit card applications before you ever got them? problem is theres two kinds of people in the world.. those that say alright no more junk mail, and those that ask, how do you do that without getting a false positive once in a while?

Re:Distrustful of Network Level Censorship

> if your going to get a false positive why filter?

My spam folder gets several hundred messages each day. It is _impossible_ for me to read every one of them to determine if it is really spam. I glance over the subject lines and read the occasional borderline one, but I _guarantee_ you that I am already getting false positives. If I dropped spamassassin and allowed the spam into my other folders I would get even more false positives as I impatiently deleted every other message as obvious spam.

Re:Distrustful of Network Level Censorship

and SPAM is WORSE, WORSE, WORSE!

If you want to receive the junk, don't use one of those services, but I fail to see how someone else choosing -to- is a problem.

Your analogy is flawed. I have a choice to use AOL|Hotmail|MSN|spamassasin|etc and I pay for the connection to download, view, respond and delete my email (not to mention the time it takes out of my day). I don't have a choice whether or not to use the USPO and it takes FAR less of my time to sort out my real mail than it does email.

If SPAM could somehow be filtered out at the router level, then I would agree with your USPO analogy and would be throwing an utter FIT. But it isn't possible (is that a web page or a webmail, is that IMAP, is that secure IMAP, is that POP3, is that email tunnelled over SSH ... no way).

Until there is legislation with -teeth- and a way for the little guy to prosecute you are not going to see many people agree with you about server side filtering.

Re:Distrustful of Network Level Censorship

Spam control with RBLs is, in fact, decentralized. There are many RBLs to choose from, and any that are too severe will not be used for long if they generate too many false positives. As a system admin, I have my choice. I use 4 RBLs right now:

• spamhaus.relays.osirusoft.com
  (this is a mirror of the Spamhaus Block List [spamhaus.org]) Well known spam operations, and is checked hourly.
• dialups.relays.osiruSoft.com
  (details at OsiruSoft [osirusoft.com]) This list is of DHCP IP addresses of home users (DSL, cable, dial up).
• dnsbl.njabl.org
  (extensive details [njabl.org] of what's on this list)
• rbl.restongeek.com
  I maintain this one myself for anything I want all my servers, primary and backup MX, to block

And there are many more [openrbl.org] to choose from. I am very happy with my results, it is a pleasure to see the reports of the mail that is blocked (see my /. journal for a sample report). If I start to think maybe one of these lists is a little too severe, or someone lets me know that there are problems with one or more of the lists, I will delete it and pick another. Or maybe not. It is my choice, I want to keep down the spam on my system, for my sake as well as my clients'.

Re:Distrustful of Network Level Censorship

> Your spam may be my correspondence -- I may want
> to get mail from those whose conduct you find
> abhorrent.

You _want_ to receive mail from the bastards that are forging my domain in their penis-enlargement ads and fake PayPal confirmation requests?

> Today, a network may responsibly be censoring
> only unwanted and unsolicited commercial e-mail.
> Next week, the powers-that-be-in-the-networks
> start censoring geek news.

I'm the only power that is on my network.

> To protect our liberties, spam control should be
> decentralized -- as close to the last mile as
> possible.

Can't get any closer to the last mile then right here in my office.

> Yes, of course, this means that the supposed
> great harm of spam -- huge volume transmissions
> through the network

"Supposed"? More than half my email is spam. And that's on a shared dialup.

my spam defense:

quite simple really:
Right here. [sigarms.com]

Great

new site called Trustic which uses a trust system

Another blacklist (with an appeals process). Run by a guy that made his millons selling eGroups to Yahoo!.

Dunno, this doesn't look too promising.

Here's my question.

Any spam measure taken at a server level could induce false positives.

I manage paid-for e-mail e-zines which I mail using PHP and sendmail (read:forged headers until I'm big enough to run my own server).

Wouldn't most server-layer anti-spam measures catch my very suspicious HTML e-zines, even if paid for?

Must be a member to appeal?

I have no interest in joining such a group. How long until they post $insanely_large_num of members as a way to try and prove the validity of their method? Bet they'll forget to mention how many members were dragged in kicking and screaming just to appeal placement on the list.

Not too impressed

Please remember that the service is beta and will start charging for advanced features once it is out of beta. As usual, worth waiting to see if it goes totally commercial. Looks like they plan to charge to allow listing multiple "trusted" servers.

A fair number of the spams I submitted came from servers that had already been voted on as TRUSTED by other users. In other words, my credability went down by reporting them as spammers.

http://www.trustic.com/ip/219.94.114.6 for example and I've got a fair number of others. Folks are either polluting the space intentionally or being very very sloppy in reporting trusted servers.

Groups like spews have a very nice evidence file, and it gets reviewed by a person. I've generally been impressed with the real community blacklist sites.

Technically the site works great and is super fast. But wouldn't follow the O'Reilly recommendation and pick it as my primary blacklist just yet (even through the guy doing the site worked with the author of the article to make changes.)

My two cents.

Just junk SMTP?

Why do n't the big players come together and come up with a better protocol instead of people trying these elaborate schemes?

Have a period where you have a parallel system going and then have a cut off time where SMTP servers die.

All it will take is the top ISP's in each country and large corporations to stop accepting SMTP mail and you'ill be sure that everyone else will then fall inline.

Or am I just being too radical?

Re:Just junk SMTP? Not Possible

Its simply too late to dump SMTP. If we would have thought about this 5 or so years ago it maybe would have been possible but now we have so many using this system its inpossible to change to a newer standard.

Just like gopher with http? You can also add a plethora of validation ontop of SMTP. SMTP, as a protocol, isn't bad. It's possible to add validation, to only accept from SMTP servers that use some sort of valid key.

Then you get to keep SMTP, and slowly migrate servers. Setup a non-profit organization for distributing SMTP authentication keys that are unique to the mail server (think SSL) and if the mail comes from that server is spam, you just block that servers key. If the server doesn't have a key, put it into a validation list or send backa response saying they need to use a mail server that supports signed-SMTP.

Easy solution, not a complete overhaul of SMTP. The problem comes in with who signs the certificates, because then you have to trust the source that delivers them. Like Verisign, et al.

Just like /.!

Sounds a bit like the /. comment moderation system!

IP banning

I know a local business that was hurt badly because the subnet that their ip addresses belonged to was added to a blackhole list. They only bought a few ip addresses and there happened to be a spammer on the same subnet. They never participated in sending spam and were never told that their ip address was blocked. Many of their emails simply did not arrive at their destinations, for no clear reason. They write and sell network security products, intended to help detect and identify hackers or even spammers looking for open relays so that they can be investigated and possibly prosecuted. This was a case where anti-spam technology hurt the near opposite of the kind of people it was meant to. I don't think they ever succeeded in getting their addresses removed from the list. All the time that went by before they knew they were on the blackhole list nearly led them to bankrupty.

Relying on RBLs

There are many problems with using RBLs to block connections. A very good description can be found here [whirlycott.com]:
I've found SpamAssassin a fairly good, rather than block messages from RBLs it analysis message content, adds points to messages in RBLs and checks known Spam databases such as Razor and Pyzor. Rules matches are given a score, and messages with a total aggregate score are tagged in the message headers, allowing users to filter these if they want to.
A main advantage of this method is that no single rule can flag a message as spam, hence legitimate mail sourcing from the badly configured mail relay has a chance of getting through, and in my mind it's probably a particularly bad idea to block any email unless it's actually addressed to you.

Or you could use a better mailer...

Here's an article talking about the details of implementing a network level spam defense with Qmail

Or, you could just use Postfix, which:

• is almost entirely compatible with sendmail. It's pretty much drop-in-and-go.
• adheres to RFCs(and there's a warning for any configuration option which would violate said RFCs)
• has builtin anti-spam tools- you can turn on, individually, any of a dozen-plus different checks, such as making sure the claimed hostname in the HELO matches the IP the connection is coming from(you can do this several ways), or that the claimed hostname matches the mail-from user@hostname(ie, if you're coming from spammer.com, you're not gonna be able to claim to be [e-mail]), etc. It's also one builtin command to check an RBL.
• has a really sharp cookie of an author(the guy wrote tcpwrapper), who isn't widely regarded as an obnoxious twit
• is completely free

Personally, I refuse to use any software written by DJB as a matter of principle. The guy flagrantly ignores RFCs because he simply feels like it and arrogantly thinks he knows better(and further that there is benefit to ignoring said RFCs).

Qmail is NOT FREE

qmail is completely free and folks that claim it isn't are just trolls.

Qmail is NOT FREE. Last I looked it was distributed without a license; now apparently it has a license, but one with oddball restrictions. If you don't believe me, do a google search with the keywords "qmail debian legal" and spend 30 minutes or so going through the various discussions.

Blackists

I run an SMTP server off my comcast cable connection... I've pretty much been learning as I go. Five weeks ago I began as a total novice, not knowing what an open relay was I spent 5 days with no authentication and as a result I was kind enough to forward some 22k messages offering investment advice. As I've learned a little more about the process... I've found ORDB and MAPS to be pretty useful and successful when it comes to blocking open relays. AOL annoys me the most, they block ranges of addresses that are dynamically allocated by ISPs and as a result I can't mail any AOL users. That's probably no big deal, I just feel descriminated against. There must be scope for a simple "Setting up your own mail server" FAQ.

When Spam Attacks!

When Spam attacks, defend yourself with a crane foot block. Then fight back with a monkey punch to its spine.

RBL's and Firewalling

I wrote a tiny little perl script that tails the maillog and firewalls (kinda teergrubes really) hosts who get a "554 Service Unavailable" more than 3 times.

I'm not coder, so it doesn't expire entries... I'm looking for someone to help make this work even better. I love the thought of causing spammers pain - and this could do that.

You can get the script from my webpage at http://www.jasonjordan.com.au [jasonjordan.com.au]

Other choice than Trustic - SPAMCOP

I have been using SPAMCop [spamcop.com] for the past 5 months at my work. I am also using QMAIL [qmail.org] as my mail server and it took me about 10 minutes to get it hooked into the Spam Cop Database. The best part it is free and it it blocks about %80 of SPAM that gets delivered - I will just have to live with the other %20. Has anyone heard of other Spam IP Databases that are available for public use?

A cheap and simple solution for a SPAM defense.

Use more mustard. Gotta drown out that gnarly taste somehow.

Dolemite
_____________________

Using Trustic with SpamPal

I use SpamPal [spampal.org] with the Bayesian filter [i-r.co.uk] as my client-side spam filter on Win2K. It works well enough but I'm always looking to improve things, so this article gave me the impetus to see if SpamPal could be made to use Trustic's DNSBL in addition to its preconfigured lists. The answer, at least for SpamPal Beta 1.295 [spampal.org], is yes--using the "Extra DNSBL Definitions" section of the Options dialog. Here are the steps I used to add Trustic to the DNSBLs used by SpamPal:

1. Create a Trustic account [trustic.com]
2. Once you've verified your registration, go to Trustic's DNS Query Information [trustic.com] page for your account and note the second DNS query address.
3. In SpamPal, open the Options dialog and drill down to the "Extra DNSBL Definitions" section. Click the "Extra DNSBL Instructions" button for information on adding a DNSBL to SpamPal. Read this text and then close the file.
4. Click the "Extra DNSBL Definitions" button. This opens "extra_dnsbl.txt". Add a new DNSBL entry as follows:

   LIST Trustic

   
   NAME Trustic DNSBL
   TYPE STANDARD
   WEBSITE http://www.trustic.com/
   ZONE queryaddress
   DESCRIPTION Trustic is a community-based block list that prevents untrusted servers from sending spam. It is a new approach to the spam problem, and it is better than existing solutions.

   Substitute the personalized query address you saw in step 2 above for queryaddress.
5. Save and close "extra_dnsbl.txt", then exit SpamPal and relaunch it.
6. Open SpamPal's Options dialog and drill down to Spam-Detection, Blacklists, Public Blacklists. Trustic should now appear on the list. Select it and click Apply, OK.

That's it--SpamPal should now be checking Trustic's DNSBL for your incoming mail. Trustic may require additional RESULT_CODE settings--I'm waiting for a response from Trustic and will follow up if needed.

IP banning is bad

Unless you have some way to identify dynamically assigned IP addresses, IP banning hits innocent parties too often. Every time Joe Sixpack, running Windows XP Home Edition on a DSL line, gets a virus that spams, the next few people to get a lease on that IP address have mail blocked.

There's got to be a better way.

A spam free world...

Is but an attitude shift away. All you have to do is follow Hotmail's idea of an exclusive address list. Nothing comes through for any individual user except what's from addresses in that user's personal address list. Keep the filtering feature on the client side, so all the mail server does is essentially route mail traffic, like any router should. Keep the processing load on the client. If the users want an email from a certain source, they're going to have to add the address in manually. A little unique cert generation during the initial mail client configuration, and you keep the email shotguns at bay. If someone has to reinstall their Operating System and thus has to regenerate a cert, set up an easy way for the 2 parties to re-exchange certs. Maybe utilize a website for this feature. Like public PKI... There's no reason not to do it this way with most new desktops approaching the 3 Ghz range. The users are going to have to take a proactive stance to spam, bottom line. No matter what legislation you push through, spammers will always find a way around any defenses we put up. Those who are aware of the nature of TCP/IP and programming know that whatever you implement, someone else can break. It would be trivial to force the end user to take control of their lack of spam, and thus break that particularly annoying 'feature' of open standards.

trustic

Ive just signed up with Trustic [trustic.com] after reading this article. Great service, plus its free. Im currently pushing all the spam SpamAssassin finds to my Trustic account with procmail, to register my negative recommendations.

This is certainly one way we can all help to fight spam.

No, it's a numbers and money game

I am the CTO at a company that provides hosted internet services, including email. We send around 3m pieces of email a week to our customers (opt-in only) lists. Speaking from the legitimate provider's viewpoint, I have a couple of observations:

1. RBL's don't work - community RBL's are used by relatively few mail systems out there; perhaps 1% of email addresses at most have RBL filtering on them at server or personal level, and the audience of any one RBL is just too small for it to have any value. Yes, using an RBL may stop *you* from receiving (some) spam, and in the short term that's all you care about, but it doesn't stop spam from being of value to the spammer. Just like the drug war, we will only win by making it unporfitable to send spam.

The biggest impact we see from RBL's is fielding individual "false-positive" complaints; we don't allow customers to send spam, so we get very few, but there's always the occasional idiot who signs up for a list and forgets, and who is too proud to click on the unsubscribe link.

What matters for delivery of my cleints' legit mailing lists, and what also a spammer cares about for delivering his spam, is delivery to the big guys - AOL, Yahoo, Hotmail, Earthlink, etc. If you're trying to email Joe Public, those guys have 50%+ of the market. Any successful spammer will have his energies focused on end-running their filters and will give a fig if RBL'ed.

2. IP-based filtering for consumer connections *does* work - ISP's and universities need to block port 25 outbound from consumer connections and desktops / 802.11 respectively. Spammers need a network connection; cut off their main source. This would stop not only transient spammers, but those who hack cable modem users.

AOL's efforts here on behalf of their users are commendable, but blocking these IP's *at source* where the blocker is making an informed decision and has the data to keep the filters accurate, is the way to go; a grassroots effort to inform ISPs about the benefits of this would be valuable.

This would leave spammers who are using business-class connections (where the ISP thus delegates the responsibility to run mail servers) which are much, much fewer in number and thus much easier to police.

Before anyone who runs their own SMTP server on tehir home Linux box cries foul, I should point out thay I do to, and I just have sendmail push everything through my ISP's SMTP relay. Big deal.

3. Money - money is they key to this. Make it uneconomic to spam, and the problem goes away.

I have one solution which I think wouls work well; like RBL's or source-end IP filtering, it suffers from the problem that it requires a large critical mass, so I think legal is the best route: I am speaking in terms of the USA, but this would work in other countries.

- anyone sending (pick a number, say 50k) pieces of email a month or more must register with the national email registry - this will cost $10k per year (this kind of price is essential to keep the spammers out, and it covers the cost of operating it). ISPs and email distributors are required both by law and defacto to sign up to be in business, and to them it's a modest cost.

- the registry will maintain an anti-spam policy and audit registrants against their track record of enforcing it; policy would need to include things like each email having clear unsubscribe info, info on where the address came from, etc.

- there will be a national "do-not-send-opt-out-mailings" list against which email marketers must clean lists which they buy; many countries have had this kind of list for phone and snail mail for quite some time, e.g. UK

- ISPs can then use the registry as a whitelist, and simply block every other IP address. Any business / individual too small to need to register can just forward their email via their upstream provider, who is then on the hook to manage their email behaviour.

Yes, it takes away some freedom to operate ones own email service, but equally I don't ru

OS flaws make technical solutions difficult

Spam is not just about sending unwanted email from rogue servers. Even if the Internet email system consisted of a 100% controlled network that excluded spammers' systems, there would be a serious spam problem. Why? More and more spam is sent from systems infected by viruses and trojans, and as other avenues get closed, this most promising one will be used to the maximum.
Let me race down the technology curve and predict some of the wonderful things that will happen in the war on spam:

- the majority of spam will originate from 'infected PCs'.
- some smart person will cause email to be charged, and millions of innocent users will get incredible invoices for email they 'never sent'
- as the number of infected PCs being remotely controlled by spammers increases, the volume sent from each PC will go random and low enough to be effectively undetectable.
- spammers will start modifying real email to attach their own messages.
- spammers will start modifying URLs in real email to point to their own websites.
- spammers will find ways to infect MSIE to do the same thing.
- anti-spam software will start to resemble anti-virus software, as spammers and virus writers hook-up into an organized (criminal) network.
- anti-spam software will be the main thing targetted by new viruses.

and all this time, 80% of PC users will remain blisfully unaware that their PCs are sending shiploads of spam around the world.

The basic problem is that the (Windows) PC is simply too complex, too connected, and too vulnerable to use as a secure communications device.

There is an answer somewhere... but I don't believe it lies in technological solutions, nor does it lie in making email paid, nor does it lie in attacking the servers and networks used to send spam. It is rather to understand that simplicity and transparency is the key to security. In the case of PCs, this means arriving at a OS/application combination that is immune to trojans and viruses, not thanks to the latest anti-virus scanners, but thanks to an inherently uncrackable design.

TMDA

My favorite solution is still TMDA [tmda.net], a free challenge-response auto-whitelist and complex filtering system for Linux. I realize you anti-challenege / response people won't hit the "R" key for me, but I consider that a useful filter...

Spam defense

Is this, like, protection for small tins of semi-edible goop? "Please, no! Take the tinned tomatoes but don't hurt my Spam!"

That ain't working, that's the way you do it ...

... Money for nothing and logs for free.

98% block rate: RBL + custom rules

80% effect - easy and low maintainance:
- several RBL sources (dsbl.org, spamcop.net, spamhaus.org, etc.)
- geographical information (china, korea, etc.)
- listings of spammy providers (XO, RR, COMCAST, ATT, UUNET) and countries (CN, KR, etc) from blackholes.us
- some netblocks semms to send spam only: 4/8 (genuity ), 12/8 (ATT), 218/8 and 61.156 (china), more to come whenever

20% maintainance required:
- add /24 or /16 netblocks of persistent spam sources (thanks to postfix for the logs :-), dialups and proxy sources
- block spammy domains (libero.it, daily-promotions.net, adelphia.net, etc.)
- use some spambait addresses (nobody has any reason to sent mail to users who left years ago or to role accounts abandoned for a long time) and add any mail senders /24 block

enjoy the 5 or 10 spams coming through per week and complain to providers if local, otherwise add to block list.

Go after the spammers

With all of these comments about how there will always be a back door for the spammers to send their mail out, doesn't it seem more feasible to go after the companies who send out the spam?

Re:Just like always...

Lol, it will give the spammers unlimited addresses by which to cover themselves, thereby eliminating the need to hijack others servers.

Or at least that is my interpretation of how IPv6 would affect spam.

Re:Just like always...

What the hell does IPv6 have to do with spam?
Well it makes it much harder to scan for servers that are vulnerable, either for hijacking or open smtp services.

Re:Just like always...

Rather a severe punishment for not following protocols I should say.

Re:Just like always...

Am I missing something here? Wouldn't a way to combat this matter is to punish those companies that benefit from the SPAM, namely the manufacturer of the pill or porn or software being shoved in our face? The same technique could be used against large scale litter, when some service pays somebody to put leaflets on car winshields, etc.