The official tests of spamfilters were done in last year's TREC conference, you can read the writeup here [uwaterloo.ca] (or pdf overview [uwaterloo.ca]).

You can duplicate those tests yourself if you download the evaluation toolkit (GPL) [uwaterloo.ca]. It's a modular system where you can add a mail corpus (either one of the public TREC ones, or you can make your own trivially), and add a spamfilter package (there are 10 or so to download from the web, or create your own as per documentation).

There's also a video talk [researchchannel.org] given at Microsoft research which should cover pretty much the same ground, if text mode is slashdotted :).

There's a new scheduled test towards the end of the year at TREC 2006.

http://popfile.sourceforge.net/ [sourceforge.net]

http://plg.uwaterloo.ca/~gvcormac/spamcormack.html / [uwaterloo.ca]

A false positive rate of 1:100

No, better than 1:100 - that's what <1% means. It's actually around the 1:500

Usually, anti-spam solutions which give more than 1:100000 are considered worthless

Got links, or is that just your opinion?

However, note that we are talking about two separate scenarios:

• a home server for an user with no responsibilities
• a project/ISP-wide mail server

Go get VideoLAN client and you can stream download the OGG version. Just open the URL as a Network Stream:

http://www.csclub.uwaterloo.ca/media/files/cormack -spam.ogg [uwaterloo.ca]

Very handy use of VLC! :)

TREC 2006 [nist.gov] evaluations are now underway [uwaterloo.ca].

While it is reasonable to conjecture that spam has changed so as to defeat spam filtering techniques, or will change so as to defeat the PPM technique that did well at TREC, the historical evidence does not support this conjecture. In particular:

• The spam filters tested in 2004 give pretty well exactly the same performance on 2005 and 2006 data.
• New versions of the filters are a little bit better, but not by leaps and bounds, and also get about the same results over the last 2.5 years of data.
• There is no evidence that "Bayesian poisining" is a viable technique for defeating statistical spam filters in anything but a very artifical laboratory environment where the poisoner has access to the recipient's inbox

Andrej Bratko used PPM -- a well-known data compression technique to compress ham and spam separately. Well actually he didn't compress them but just build the statistical model necessary to compress them. Then he simply (tentatively) added the unknown message to each model and chose the one that compressed it best. The general technique of using compression has been mentioned here and elsewhere but Bratko used a much stronger compression scheme and was somewhat clever about it.

I later reproduced Bratko's results using DMC -- a compression schem that I invented 20 years ago -- and got some interesting results. We have a journal article in press describing it and also an evaluation paper at CEAS 2006 [www.ceas.cc].

Bratko A., Cormack G. V., Filipic B., Lynam T. R. and Zupan B., Spam Filtering Using Statistical Data Compression Models [uwaterloo.ca]

Fidelis Assis (who has now gone solo after having participated in the CRM114 project) shows great results for his recent solo effort: OSBF-lua [luaforge.net] Bratko's PPM spam filter [ai.ijs.si] -- the one that did great at TREC -- is not yet packaged as a drop-in filter. Same for my DMC spam filter [www.ceas.cc].

The actual TREC 2005 tests referred to in TFA are here. [uwaterloo.ca]

Personally, DSPAM isn't nearly as good and has flagged many legitamate messages and sent them to the Junk folder.

And what happened when you retrained those false positives as ham? Did you see future mails of the same/similar type get caught again? I bet you didn't.

I've been using dspam for a very long time for my users, and they love it. They love having zero spam in their mailbox, they love the simplicity of the user interface. They love how it treats users on a per-user basis, not globally (i.e. some users WANT html emails, some do not. Each can mark them as they see fit.)

Here's an example of my own stats..

hacker: TP True Positives: 122601
TN True Negatives: 124711
FP False Positives: 211
FN False Negatives: 1046
SC Spam Corpusfed: 3708
NC Nonspam Corpusfed: 456
TL Training Left: 0
SHR Spam Hit Rate 99.15%
HSR Ham Strike Rate: 0.17%
OCA Overall Accuracy: 99.49%

• Spamhaus SBL/XBL filtering (hard SMTP-time DNSBLing) based on my expereince with them and their consistent listing of VIOLATORS, not just anyone who shares a netblock with a spammer (i.e. they may not catch as much as some others, but they don't have the FP rate that others do)
  
• Greylisting. This is controversial because many people can't tolerate the delay it introduces. I found a radical decrease in spam when using it (because honeypots have already located a spammer by the time they try again), and only marginal headaches introduced by the delays of new senders. YMMV, and I wouldn't use it in a production environment.
  
• SpamAssassin. I tweek the RBL settings (I *never* want to even score SORBS, for example), and configure razor, but otherwise pretty much leave it in its default configuration, and it works great!
  
• Thunderbird mail filtering. I use evolution and thunderbird. I don't bother turning on mail filtering in evolution, since it uses SpamAssassin, and there's no point using SA twice on the same message. I *do* use thunderbirds filtering as yet-another layer of filtering when I'm using that, and it does a good job of classifying what little spam is left.