Proving Which Spam Filters work Best 263
pirateninja writes "Dr. Gord Cormack decided to find and prove what the best spam filter is. In his study he looked at the major spam filters (DSPAM, SpamAssassin, etc.) along with those submitted by various academics. The results are quite surprising, with a previously unheard-of spam filter, which uses ideas from various compression algorithms, performing the best overall. He recently presented the results and methodology used in a presentation titled 'Spam Filters, Do they Work? and Can you prove it?'" Note that this is a video of his presentation.
Not at 400 (Score:0, Informative)
In my experience... (Score:5, Informative)
Re:Why not just douse the server in gas... (Score:2, Informative)
Re:Combo of SpamAssassin and Spamhaus (Score:3, Informative)
Got to go with Brightmail (Score:5, Informative)
I also echo a gripe of other posters. Its nice to have a video but 500MB video file it a bit much. A 50KB pie chart or bar graph would have been nice.
Flaw in the test (Score:5, Informative)
As with most choices like this, factors such as ease of use, speed, and resource efficiency can overshadow selectivity. No system is perfect, so it's perfectly reasonable to go with a system that's pretty good if you already are using it, rather than switching to the latest cool thing.
I have found that using two dissimilar systems in a chain is quite effective.
text versions of the material (Score:5, Informative)
The official tests of spamfilters were done in last year's TREC conference, you can read the writeup here [uwaterloo.ca] (or pdf overview [uwaterloo.ca]).
You can duplicate those tests yourself if you download the evaluation toolkit (GPL) [uwaterloo.ca]. It's a modular system where you can add a mail corpus (either one of the public TREC ones, or you can make your own trivially), and add a spamfilter package (there are 10 or so to download from the web, or create your own as per documentation).
There's also a video talk [researchchannel.org] given at Microsoft research which should cover pretty much the same ground, if text mode is slashdotted :).
There's a new scheduled test towards the end of the year at TREC 2006.
Re:I have one word: (Score:3, Informative)
Their main problem is the system doesn't learn. Using their web interface, I look through the spam folder and request delivery of all the false positives. The next day, nearly-identical mails are still generating false positives. You'd think it would be easy these days to design a filter that learns from negative reinforcement.
Re:In my experience... (Score:5, Informative)
http://popfile.sourceforge.net/ [sourceforge.net]
Possible Text Version (Score:4, Informative)
Gordon Cormack and Thomas Lynam
Full Text, May 29, 2006 - PDF Format
http://plg.uwaterloo.ca/~gvcormac/spamcormack.html / [uwaterloo.ca]
Out of Date and Worthless (Score:5, Informative)
He tested spamassassin 2.3 - that's ancient! I'd imagine the other tools are similarly obsolete.
We currently use SA 3.1.4 with a well-trained Bayes database and Razor, Pyzor, and DCC.
Throw in a few custom rules and a selection of rules from http://www.rulesemporium.com/ [rulesemporium.com] and the results are outstanding.
With the new sa-update feature the core rules are updated between point releases, which came in useful this week dealing with the new image spams which seemed to be designed to avoid detection by spamassassin. Thanks Theo.
And the folk on the spamassassin-users mailing list really rock.
Re:MS Anti Spam... (Score:3, Informative)
However, note that we are talking about two separate scenarios:
Re:Why do they try? (Score:1, Informative)
Not making any judgements but the "clued" category includes Gmail, Yahoo Mail,
AOL, corporate IT managers and university mail server admins.
Torrent (Score:4, Informative)
Dspam floats my boat (Score:3, Informative)
Re:Torrent (Score:2, Informative)
Go get VideoLAN client and you can stream download the OGG version. Just open the URL as a Network Stream:
http://www.csclub.uwaterloo.ca/media/files/cormack -spam.ogg [uwaterloo.ca]
Very handy use of VLC! :)
Re:Out of Date and Worthless (Score:4, Informative)
TREC 2006 [nist.gov] evaluations are now underway [uwaterloo.ca].
While it is reasonable to conjecture that spam has changed so as to defeat spam filtering techniques, or will change so as to defeat the PPM technique that did well at TREC, the historical evidence does not support this conjecture. In particular:
Re:Possible Text Version (Score:4, Informative)
Fidelis Assis (who has now gone solo after having participated in the CRM114 project) shows great results for his recent solo effort: OSBF-lua [luaforge.net] Bratko's PPM spam filter [ai.ijs.si] -- the one that did great at TREC -- is not yet packaged as a drop-in filter. Same for my DMC spam filter [www.ceas.cc].
The actual TREC 2005 tests referred to in TFA are here. [uwaterloo.ca]
Slides from the presentation (Score:3, Informative)
Re:Got to go with Brightmail (Score:3, Informative)
And what happened when you retrained those false positives as ham? Did you see future mails of the same/similar type get caught again? I bet you didn't.
I've been using dspam for a very long time for my users, and they love it. They love having zero spam in their mailbox, they love the simplicity of the user interface. They love how it treats users on a per-user basis, not globally (i.e. some users WANT html emails, some do not. Each can mark them as they see fit.)
Here's an example of my own stats..
hacker: TP True Positives: 122601
TN True Negatives: 124711
FP False Positives: 211
FN False Negatives: 1046
SC Spam Corpusfed: 3708
NC Nonspam Corpusfed: 456
TL Training Left: 0
SHR Spam Hit Rate 99.15%
HSR Ham Strike Rate: 0.17%
OCA Overall Accuracy: 99.49%
Re:In my experience... (Score:3, Informative)
YMMV. Good luck.
Re:Torrent (Score:3, Informative)
Re:Harder! (Score:2, Informative)
[/innocence]
Here are UW's traffic stats, in case anyone's interested:
http://noc.uwaterloo.ca/cgi-bin/14all.cgi?log=cn-
Also note the spikes on Monday and Tuesday from when we posted our last [slashdot.org] two [slashdot.org] talks.
Re:Harder! (Score:3, Informative)
Excellent 'In Our Time' programme on Babylon and it's Literature here - http://www.bbc.co.uk/radio4/history/inourtime/ino