More on Bayesian Spam Filtering

michaeld writes "The "Bayesian" techniques for spam filtering recently publicized in Paul Graham's essay A Plan for Spam doesn't actually seem to have anything Bayesian about it, according to Gary Robinson (an expert on collaborative filtering). It is based on a non-Bayesian probabilistic approach. It works well enough, because it is frequently the case that technology doesn't have to be 100% perfect in order to do something that really needs to be done. The problem interested Robinson, and he posted his thoughts about trying to fix the problems in the Graham approach, including adding an actual Bayesian element to the calculations."

How about Macchiavellian Spam Filtering

kill 'em. might = right

Spam spam spam

Well I guess spam comes in different size tins sometimes, and with different labels so you can tell the spam apart. I like Hot and Spicy Spam. Mmmm.

Of course, the 1% of non-spam that accidentally gets filtered out is just collateral damage (except it's normally something really important like a tin of processed peas or something).

I'm going to sit down now and take some more HGH.

spam

Someone came up with this idea recently, and I like it, so I've been repeating it. Instead of illegalizing spam, which i would love if it worked, but it won't, require spammers to indicate the nature of the email--anonymous, commercial, with a word or such in the subject line, which can then be filtered by individual recipients according to their desires. It would not be as free-speech-limiting as banning spam, and spam would die out due to ineffectiveness once most everyone filtered it.

I still think passive euthanasia is the best way.

Why is such a simple problem that pisses off 99.9% of the population is so hard to manage on a global scale? I mean, EVERYONE is pissed off at getting spammed, everyone would LOVE legislation to sodomize local spammer with a baseball bat, oversea is a different problem but country/continent-wide spam is 1/2 of my problem and can be easily be taken care of with proper legislation. For once a restrictive legislation would get 99% support... you don't see that everyday. like I mentionned before, I don't get our politicians, they say they work for us, they try to find clever ways to tax us, remove control that we used to have and all, but something on which they would get unprecedented support, they are simply sitting on the issue...

Until politicians will be fed up and people will actually get SUED for spamming (for once you could have a good reason to sue real bad guys) nothing will change.

Yes I know in SOME states it's beginning, so for local spam in a few years from now I think legislation will make it's way and we'll be able to look in our mailbox and stop having TD waterhouse spamming when you already have an account with them, etc.

The other problem now is oversea spamming, especially coming from China/Taiwan. I mean.. I don't read chineese, I don't plan on buying that #.#" something oversea, so why do they spam us like that? I never get it, but I'd be all for passive euthanasia (i.e. ban their IP at router level) and if this is bad for buisness or relations or whatever, well MAYBE they will do something about it.

Here where I work, it's simple, one spam, I ban a whole class straight off the servers, if one day I get a call because someone couldn't reach us (if they really need to reach us, we have a phone anyways!) I'll be sure to mention him Why. too bad this is not happening at the backbone level, because some people would get their act together fast and apply a legislation globally.

Tutorial on Bayesian Inference

The timing of this article seems impecable, since I am myself trying to learn about Bayesian Statistics.

I am a Computer Science student [ime.usp.br] studying Computational Biology [ime.usp.br] (more specifically, Sequence Alignments) and while I have a bit of background on Classical Statistics, I was (and still am) completely ignorant about Bayesian Statistics.

It is only now that I'm trying to learn about Hidden Markov Models and its applications to Sequence Alignment that Ifinally decided to learn the basic hypothesis about Bayesian Statistics and how it differs from the hypothesis made by the Classical Statistics.

During my searches for finding introductory material on Bayesian Statistics, I found this course page [arizona.edu] which has some nice introductory notes, including Bayesian Statistics.

I hope that other people find this resource as useful as I did.

Post your results here

I'd like to head the results of anyone who has implemented one of these probabilistic filtering systems. I implemented a modifed version of Paul Graham's system and so far it kicks ass. So far it has trapped over 600 spams without any false positives. I receive almost 100 spams a day and over the last week I have generally only had to delete one or two by hand. The rest go directly to jail.

I'd like to hear about modifications to this system. I removed Graham's doubling of "good" word frequencies, and I trained my filter using digrams. I also tried all the various methods supplied by the program "rainbow", with good results, but the implmentation was too slow and klunky to place in the middle of my email delivery system. What are other possible modifications?

Re:Post your results here

I hacked it together in Perl, to make use of the Berkeley DB interfaces and the MIME parsing modules. Took about 30 minutes. I'm working on a C library that could be linked into mutt or pine or whatever, but I'm finding the available MIME code in C cumbersome.

You can grab the source here [saturn5.com], but it is specific to the exact way that my mail gets delivered (via offlineimap into maildirs).

The proof of the pudding...

...is in the eating. I think the same applies to spam. Paul showed, to his satisfaction, that the technique he used worked for his samples. Gary proposes some changes that would improve the filter's accuracy, but does not test these theories.

We will now have many slashdot posts saying "I've not tested this but I think A (or B, or C, or X)"

Here's where the scientific method comes into its own. Anyone who cares enough can actually test and post their results. I'd be interested in seeing what they look like. I don't have a database of spam to test against (and please don't volunteer to sign me up for some :) but it would be interesting to see whether what looks convincing in theory pays off in practice.

poor Hotmail users are still in the cold...

I have some tricks for Hotmail users who cannot benefit from the technique above:
Filter any message without the @ in the address.
Filter Britney, Boobs, Penis, Inches, WIN, ___ ..... and your own email address userid.
Now you only have about 40 spams a day to deal with instead of 100.
Uncheck your information from being in the MSN directory too.

Enjoy :-)
John

Terrible Spam Filters

It's good that work is being done to make a good weigted spam filter.

It's funny how bad the standard Microsoft spam filter is (the one present in outlook). It's simply a word lookup, where if the word is present the message is marked as spam. It looks for things like "for free?". You can see the full list here [iirusa.com], near the bottom. It's a little old, but not outdated (I think you can upgrade your spam filters, but I tested these, and the ones I tested work).

The adult filter isn't any better.

Naive Bayesian Learning

Finally it is worth mentioning that if you really want to go a 100% Bayesian route, try Charles Elkan's paper, "Naive Bayesian Learning". That approach should work very well but is a good deal more complicated than the approach described above.

Here is the article [nec.com][citeseer.nj.nec.com]

Let's see

P (This is spam) = P (This is Spam | It will enlarge my penis) * P (It will enlarge my penis)

Now, given that I have prior knowledge that:
P (It will enlarge my penis)

is very low,

and given that, having never encountered anything which enlarges my penis in any permanent way, I have no knowledge of
P (This is Spam | It will enlarge my penis)

and we have the product of one probability which I know is low, and another of which I have no posterior knowledge, so we conclude that P (It is Spam) is also low, and that I must have requested more information on their new penile enlargement technique.

So, that message goes into the keepers.

Meanwhile,

P (It is Spam) = P (It is Spam | Frank is getting maried) * P (Frank is getting married)

So, I know frank is getting married, since he sent me this e-mail I'm considering filtering as Spam, and weather or not it is spam is pretty much independent of whether or not frank is getting married, so.... it's Spam. Away it goes.

P.S. I've deliberated made a hash of this for a joke. The actual rule is:

P (A & B) = P (A | B) * P (B)

Whatever Jaguar (Mac OS X 10.2) uses works!

Is this what the new Mail.app in Mac OS X 10.2 uses?

I, myself, am not sure but the new Mail.app is smart and it does learn. After a week of "learning" it has correcly determined messages as spam more than 99 out of a 100 times.

filtering not the answer - maybe this is

Sadly, unless you are an ISP or other mail service provider, filtering does nothing. The spammers work in volume. They count on hitting everyone to reach that .1% that will respond. That response is what they are after and what they get paid for. You likely know better than to ever deal with anyone who spams you or to ever respond to their spam. Filtering your own e-mail has absolutely no effect on the spammer, you were not going to respond anyway. By the time you filter they have already wasted your bandidth, and perhaps mailbox capacity and even forwarding limits from a forwarding service. Your filtering is useless, puny human!

Here is a suggestion for something that might make an impact on spammers: IF I open my firewall, I see several attempts a day from people trying to get into my mail server. Of course, I don't have a mail server, but spammers are always looking for open relay points they can spam from. My suggestion: Give the a nice open relay server they can send mail to. Of course, you don't want to piss off your service provider by sending spam, and your upstream speed might limit you to less than you can receive, so rather than run a full mail server lets modify some mail server code to just accept mail and send it to the bit bucket. Maybe we can even misconfigure existing code to do this with no programming changes.

No valid user will be affected, assuming you don't otherwise run a mail server. All that bandwidth you pay for can be used to receive e-mail from spammers before it ever goes out. Eventually their customers will see the response go from .1% to 0% and their business will dry up. This will impact spammers, blocking your own spam after it's been delivered will not.

This need not even impact your own bandwidth. You can run the server when you are done using your system (Might make a nice screen saver - a black screen that just shows how many spammed addresses were prevented from getting spammed). Or you cam impose limits on bandwidth at a firewall or router, or even restrict hours of access.

If we set up enough different false open relay servers I think we could have a real impact on the spammers.

Re:filtering not the answer - maybe SPOOFSERVERS

I'm fairly sure a false relay won't work. Just like snail mail list sellers, the spammers salt their victim lists with their own valid addresses that they can check to see if the message is getting out.

BUT, an early spam filter at an ISP worked just like that. The design parameters were 1) that spam filtering require no more resources than actual delivery of the message, and 2) the filter give no indication to the spammer that the message was not going to delivered. This gives the spammer no feedback and forces THEM to waste CPU cycles which will slow them down.

Neural Net Spam Filtering

At UCSD, Bob Boyer and I wrote a neural net spam filter. Neural Nets, as everyone knows, are not really like biological brains, but really just statistical engines similar to the approach the guy above claimed to do.

Our approach worked pretty well (95-97% accuracy), and we had to deal with the same issues that the above "Bayesian" approach did. I.e., weighing the neurons so that false positives occur much less frequently than false negatives, etc. We built it using data on spam collected from the UCI machine learning repository.

It ties in with procmail. I'm not really a windows guy, so if anyone knows how to put a filter between an IMAP server and Microsoft Outlook/Netscape Communicator, I'd be interested in hearing how it's done.

The README for it is at: http://www-cse.ucsd.edu/~wkerney/spamfilter.README
And you can download it at:
http://www-cse.ucsd.edu/~wkerney/spamfilter.t ar.gz

-Bill Kerney
wkerney at ucsd.edu

SpamAssassin - duh

SpamAssassin [taint.org] works great for me. It eats about 90% of my spam, you just hack up a little procmail file for it, and you're done.

With so many people using SpamAssassin these days, I can't see how this is a timely or newsworthy item. More like from the been-there-done-that-dept..

How do you pronounce "Bayesian" anyways?

While I love everything there is to love about open source (code and ideas), I kind of worry when I read how successful all these new Bayesian/Grahamian filtering techniques work.

Not being a coder or statistician myself, I'm left wondering if the spammers can exploit it for a workaround. Is there something "built in" to these filtering techniques that can be used by spammers to effectively circumvent them?

Well...

I hate to give any kind of credit to M$ but they patented the idea of using Bayesian analysis for spam filtering circa 1995. They even had it in one of thier beta's. However the filters were tagging some of those fricking Blue Mountain greeting cards as spam (imagine that!) so Blue Mountain sued them on anti-competitive grounds and M$ pulled it. Blue Mountain wanted to have the spam filters universally pass Blue Mountain content but MS refused that on the grounds that if a user considers it spam then it is in fact spam to them (Hurray for the "bad guys"!). The law suit has been settled/dropped/died for reasons I don't know.

Anyway I hear that the next version of MSN will have a Bayesian filter and that it will be introduced in an up coming version of Outlook Express (no idea about Exchange and Outlook).

BTW I believe internally MS uses this technique for spam control and that they don't seem to have any spam problems.

Why just spam?

Sure, spam is a big problem, but right now only 10-20% of my emails are spam, and most are easily identifiable by subject.

On the other hand, I get hundreds of emails every few days covering a range of topics, which need to be manually sorted into folders.

What I'd like to see, and I suspect I'm not alone here, is similar software that can sort email into any number of categories, not just spam and non-spam.

For example, if I have an email folder called 'fishing', containg emails from fishing buddies, then next time I get an email containg references to 'casting', 'trout' and 'it was *this* long', it should be sorted into that folder automatically.

I'd be curious to know if there's any existing software to do this, and if not, I'd be tempted to have a go at knocking something up to do this.

One tricky bit would be how to integrate it with the email client. I'd imagine that users wouldn't want to switch away from Outlook/Mozilla/Mutt/Whatever merely for this feature, so it would have to be client-agnostic.

I'm thinking that implementing a simple IMAP server would be the easiest option since this allows for server-side folder management. It would then be case of maintaining word counts (Bayesian or otherwise) for each folder, and classifying mail accordingly.

Anyone else had any thoughts along these lines?

Brain exploded

Note to statisticians: the product of the probabilities is monotonic with the Fisher inverse chi-square combined probability technique from meta-analysis. The null hypothesis is that the probabilities are independent and uniformly distributed.

Ouch! My brain is hurting, Doc!

Bayesian Filtering Works

I have implemented Paul Graham's algorithm at my corporation, and it is blocking 90-97% of our spam each day. It is "good stuff". Combine that with Razor v2 and some other filtering I do, and nary a spam gets thru.

Could someone tell me...

... what exactly bayesian means?

authorization based email box

I realized one day that filtering spam out by content is a futile exercise. I use a simple method that has worked perfectly: If the FROM address of an incoming message is not in my contact list, the message is Trashed. Before emptying the trash, I'll glance through it to be sure that I didn't recieve a legitimate message from someone not in my list. Since I've used this, not one spam has ever appeared in my Inbox. This is important since I use mobile devices and other strange ways to access my email that would be very sensitive to spam overload. Fact is, 99.999% of email I receive is either 1.) From people already on my contact list, or 2.) People who inform me they're going to send an email. Before I give out my address, I inform them that I need to know their address first, and add it to my contact list. If someone gets my email from someone other than me, or otherwise didn't talk to me first, I probably don't want their email anyway. And if it's important, they'll get in touch with me. I'm using Outlook for this solution and use a rule that moves all the messages out of the Inbox that don't meet this criteria. I plan to switch to Evolution soon under Mandrake and I'm sure I can program a similar function. It's much easier to spot 1 message from a legitimate sender out of 100 spams (takes only a few seconds in fact) than it takes to manually delete spams or constantly fiddle with filters. Each day, I'll glance at the list of 100-200 spams that have collected in my trash box, and within a few seconds, I can spot if someone I know has sent me something who isn't in my list. From that point forward, they're in my contact list, and it never happens again. At some point I plan to set up an auto-reply system that gives people a URL that they can visit to "ask for permission" to send me email. Spammers won't use it. I haven't bothered yet because I'll need to carefully design this to prevent my address from being "confirmed" by spammers as a result of this message, but I have ideas for that (send from a null account, use a picture of my email address in the message, with instructions on how to ask permission.) At that point, I can safely instant-trash all unrecognized recipients. I'd love some feedback on this method. It's worked great for me, though admittedly it won't work for those who recieve many emails from new contacts, such as someone who publishes (eek!) their address on a site for inviting new messages.

keyword matching isnt the answer

sites like yahoo, hotmail, etc are in a unique position to rid their users of spam.

i don't see why they cant implement some system that scans incoming mail for its users' mailboxes, maybe does a checksum for each message or something, and if it finds that a number of its users are receiving exactly (or nearly exactly) the same message, assume it's spam. nuke the messages, and any new incoming ones.

yeah, if such a system only scans a small number of mailboxes, it may filter out mailing list posts and so on. but it gets more and more reliable the higher number of mailboxes it tracks.

this avoids searching for certain keywords and eliminates false positives. after all, how well would these keyword searching methods do if i were to quote a spam message in an email to a friend?

Bayesian vs not isn't really the point

Gary is both right in some respects and irrelevant in others. Here's the key line in his article that deflates it a bit:

It is untested as of now. It is based purely on theoretical reasoning. If anyone wants to try and it test it in comparison to other techniques, I'd be very interested in hearing the outcome.

On the other hand Paul Graham has actually tested his model and it works. I've worked it up in perl and tested it on my own data set and it works there, too. Paul acknowledges that he's being a bit fast and dirty, but the proof is in the pudding. The rest is just academic quibbling over the fine points.

I'm not sure why this particular article needed to be posted, as it's just one of several alternative approaches and an untested one at that. On Paul's page, he also lists several published academic papers with other alternatives -- all actually tested, of course.

Gary is basically right in questioning the use of the word "Bayesian". Paul's approach is more about weighing "evidence" as given by the appearance of certain words, rather than in figuring out the probability of spam assuming a "prior". See Paul's explanation [paulgraham.com], but if you check the article he references at the end, you'll note that the method Paul uses is only one of several methods to solve an underspecified problems. It's a reasonable guess, not necessarily the only guess.

Looking at another article [lanl.gov] Paul references, given the word independence assumption, the more formal Naive Bayesian approach calculates as follows:
p(spam) = [ p(spam)*p(word1|spam)*...*p(wordn|spam) ] / [ p(spam)*p(word1|spam)*...*p(wordn|spam) + p(!spam)*p(word1|!spam)*...*p(wordn|!spam)]

This is similar to Paul's approach except for including a "prior" assumption of p(spam) -- the expected probability of any email being spam, calcuated from the historically observed frequency of spam. By leaving it out, Paul implicitly assumes that 50% of mail is spam -- that's his "prior" estimate of the spam rate. Given the other adjustments he makes to his sample, that appears to be acceptable in practice. (Paul overweights the spam prior, but also overweights the effects of "good" words.)

I'd personally prefer to overweight the "good" e-mails entirely rather than just put a "good-multiplier" on them like Paul does, but that's just quibbling over small bits.

As to the bit that Gary raises about Paul assuming a spam probability for an unknown word -- Paul originally said .2, then revised to .4, but really should have put it at .5 or just excluded it from all calculations. A new word has no robustness as a predictor (which is why Paul dropped words that didn't appear five times anyway). In practice, a new word at .4 isn't going to be among the 15 most interesting words to make the calculation from, anyway.

-XDG

not 100% - not good enough

because it is frequently the case that technology doesn't have to be 100% perfect in order to do something that really needs to be done

Right. Try that one again after your non-100% effective filter starts filtering out business e-mails. Then where'll ya be? nowhere.

AI people have absolutely no common sense. Its been proven by my neural net.

Discussion in comp.lang.python

There was extensive discussion of Graham's spam filtering algorithm and potential improvements on comp.lang.python [comp.lang.python] in mid-to-late August. Check Google Groups [google.com] for the subjects "Lisp to Python translation criticism?" and "Graham's spam filter."

How long until we throw out the current e-mail sys

How long until we throw out the current e-mail system.

I own my own domain, which makes it easier, but we really need a system designed to filter. And make it easier. This is my uninformed proposal. Perhaps it won't work, but it seems something is needed.

People should have a private/public e-mail address. They should all go the same "account" and be part of the basic plan for any e-mail user.

privateauthentication~[e-mail]

I know this is important and relevant

publicauthentication~[e-mail]

I gave this person my e-mail address

[e-mail] will go into the crap bin and be deleted eventually. Perhaps some program could be used to alert users of possible important mail pieces there.

Then we could also have some system to CHANGE the private authentication or public authentication that is form based. I.e. This address has been disconnected. Please apply for the new password.

Dictionary spam?

It seems to me a countermeasure spammers might try is including a dictionary with their spam. Since filters are for sure going to be conservative and avoid false positives, they'll latch onto "good" words from the dictionary and ignore "bad" words from the spam.

It works well, but some spammers circumvent it

There I was on vacation, wondering what to do with my free time, and a spam popped into my inbox. I remembered the article about Graham's statistical technique, which seemed a lot more interesting than an arbitrary keyword list or a set of ad-hoc rules, so I decided to write an anti-spam program. Vacation accomplished.

After a couple of weeks I've built up a big enough spambase that Graham's algorithm is pretty close to 100% effective (and no false positives at all).

However, I did run into one problem: Some particularly devious spammers are base64 encoding their email so that it can't be scanned by programs like this. (I can't think of any other reason why they're using base64 encoding for text/plain or text/html messages.)

After I added code to check the email header and decode the message body it worked much better.

Jaguar

Apple's new spam detector works amazingly well for me. After some initial jitters it pretty much never gets false positives these days.

microsofts trademark

why are we even considering this method when microsoft has a trademark on it? nothing can be done.. they have a lock on it. trademark here [uspto.gov]

How is this working exactly?

I think it might me interesting to apply AI methods in fighting spam, especially machine learning. For example, you could have a spam filter that is able to learn. You just show 100 spam mails to the filter program, then 100 non-spam mails, and the system "learns" how spam looks like.

How does Apple's mail spamfilter work?

All it says in the help is that it is adaptive and trains itself on your previous spam. It would be nice to see some source... and be able to patch it if we don't like it.... oh well, whining won't get me anywhere.

A call for suggestions, and coders...

Let me start by saying I know very little about coding, otherwise I'd probably already be rushing off to a night of coding by the glow from my monitor.

When the first Bayesian spam filtering article was posted, I thought it was a great idea, and this article just reinforces that idea. However, it would be interesting to build some sort of Sendmail module (or whatever MTA you like), but add some additional functionality:

1. Option to return a 550 error if the message is determined to be spam: "550 Delivery blocked; Bayesian filter reports spam probability of nn%"
- Right before reporting this error, wait n seconds or alternately, slow connection to n bps for n minutes.
- After reporting the error, "deliver" the Subject and Body of the email to the spam words database.
2. Inclusion of a whitelist, by IP, reverse DNS, MAIL FROM address, or RCPT TO address, header To: address, header From: address, etc.
3. Configuration of account where spams can be forwarded to, for automatic addition to the database.
- Perhaps this could be combined with the blacklist/whitelist. For example, any emails to [e-mail] are always added to the DB. The entry could be as follows (similar to the Sendmail access map):
[e-mail] <tab> BAYESIAN:SILENT
- This would allow for either silent addition to the filter (sender thinks mail was delivered -- good for spam harvesting emails, or for users to send their spam to), or a more "vocal" addition much like item #1 above, where a 550 error is reported... eg, BAYESIAN:550 or perhaps BAYESIAN:REJECT

I realize this would block a lot of mail, but I have my Sendmail currently configured to actually block spam (or what it considers spam) and have had very few issues with valid messages bouncing. Obviously, results may vary, but I'm a firm believer in rejecting spam during the SMTP conversation, not accepting it and then deleting it silently.

Does anyone else have any suggestions?

Already Patented by Microsoft...

This whole methodology is already patented by Microsoft. ANY implementation not licensed by Microsoft is going to be a violation... And now that you know, it is treble damages...

patent 6,161,130 [uspto.gov]

How is this working exactly?

I think it might me interesting to apply AI methods in fighting spam, especially machine learning. For example, you could have a spam filter that is able to learn. You just show 100 spam mails to the filter program, then 100 non-spam mails, and the system "learns" how spam looks like (maybe reinforcement learning?)

Bayesian filtering software

Seems like everyone jumped on the bandwagon and implemented a bayesian spam filter shortly after Graham's article hit the net. Best part is, theory or not, the damn thing actually works.

Paul's article lists a few of the bayesian spam filters, but here's a short list of the ones I've tried:
Gary Arnold's bayespam [garyarnold.com] is implemented in perl and geared towards qmail using maildir storage.

Brian Burton's spamprobe [sourceforge.net], written in C++, tries to remember already-seen messages, so that you can dump your spams/good mails on separate folders, have spamprobe learn from them, and delete them afterwards. Spamprobe remembers which ones it already processed, and won't reprocess a message if it's already seen it.

Eric Raymond's bogofilter is a typical ESR tool: concise, with a baroquely written man page, and quite simplistic, but does its job and does it well. ESR even uses some funny terms, like "spamicity", and "ham" (the opposite of spam). I don't like its dependency on the Judy libraries for dynamic arrays but what the heck.

Matthew Walker's BayesSpam [squirrelmail.org] plugin for Squirrelmail provides squirrelmail users with bayesian spam filtering capabilities, no longer restricting use of the technique to those with access to procmail/mailfilter systems.

Download it.

You can download the source here [garyarnold.com] if you like.

It's not from the same guy, but it's definitely derivative work.

Any linux-based POP "proxies" for this?

This is basically an "ask slashdot" question.

I have my popmail hosted by my ISP. I usually check my mail from my windows box. I'd like to configure my Linux box to periodically pull the POP3 mail from the server, spam-filter it, and then act as a "local" POP server that I'd just point my windows Eudora at.

Anyone have an easy (relatively speaking) means of doing this? Seems like each of the 3 parts (Getting mail from ISP, filter, and being a POP server) are trivial, but anything out there that would do all this or pieces that play well together?

I'm not keen on trying to deal with SMTP right now. My internet connection is a little too flaky for that...

Thanks for any ideas.

obligatory biology humor

So, a prior article described a method of spam detection which claimed to use something like Bayesian methods, and now we read that it didn't. Sounds like just another case of ...

Bayesian Mimicry

(Don't clap, just throw money.)

Method seems easily breakable.

IIUC, The proposed method normalizes (with Ln norm) over the number of words, for "spammishness" and "unspammishness" of words, combining the results.

whats stoping the spammers from attaching, say, a random scientific article longer than the spam at the end of the spam message ? This will give the spam a high grade in these bayesian method in general, but more so with his normalizing metric.

This is a reinvention...

Of classic "probabilistic searching" from the field of information retrieval. Here's a typical tutorial [vt.edu] You can feed key words from this to google to find more if you want to.

The application to spam filtering is trivial. Simply take a document set (your inbox for a month), identify the spam set (manually) and the algorithm will generate term weightings for you.

Then apply these term weightings to previous unclassified records (emails) and BINGO!

BugBear

Mozilla

Some work is being done to implement something like this in Mozilla, check bug #163188 in Bugzilla (http://bugzilla.mozilla.org).

Andrea

filtering on qmail...

I've been using and testing bayespam [garyarnold.com] (thanks Gary!) for the last week or so and am impressed by how accurate it is. Easy to install too. All of the other anti-spam tools (blackhole, spamassassin etc) are a complete nightmare to setup and configure. Obviously speed is important but I'm going to use bayespam on a case by case basis rather than filter all and any. If a user has problems, start filtering... Must remember to keep saving up that spam for my corpus. To me it doesn't matter if it *really* is bayesian or not, it works. Hope someone sorts out a Mozilla setup too...

Anyone tried TMDA?

I lkie the soun of this one - seems like it should eliminate all false positives sent by real peope and all false negatives. I worry about auto-responders and auto-reminders, though. TMDA (Tagged Message Delivery Agent) [tmda.net]

The best filter I've used

.. is Bayesspam 2.x [squirrelmail.org] for Squirrelmail [squirrelmail.org]. Its an easily installable plugin for a php-based webmail system, that uses MySQL to store the Bayesian corpus. It's also got options to limit the size of the messages to be filtered, and displays the spam probability and the 'mark as spam/nonspam' links in each email header.

Re:What happened to IBM/Redhat Article

Turned out to be a duplicate. Got pulled.

Re:But what about pr0n spam?

I know, what would we do without our free porn, and porn passwords?

Re:But what about pr0n spam?

In other words what we need is a sort of "facial recognition system."

facial?

Re:spam is already keeping up?

You must read Paul Graham's article closely. It is still identifiable as spam cuz of the phrases. As for AOL dorks sending out emails like that, I hope you don't have friends who send emails like that to you. If you are including header data in your checking routine too, your friends will also get excluded since they'll have nonspam emails in the training data.

Re:Why filter spam?

Maybe we can feed the spammers' children.
Hopefully to something large and hungry.

Re:best method .. is ..to ..

I agree, and I have a more detailed discussion of this going on somewhere below, perhaps you can come and jump on my side, since I'm getting flamed down there. :)

Re:Why is spam still a problem?

I disagree with you. I have a more thorough thread on this below you might want to check out.

My position is that there is already an authorization step for virtually all senders of email - you need to get the recipients address.

Most people are careful not to publish their address for obvious reasons. To get someones address, you have to ask them in person somehow.

I use this system, and I haven't had any problems when I ask someone "by the way, what address will you be sending from? I need to add you to my list." Most don't ask why since most people don't ask for someone's address and refuse to give their own - it's stupid anyway, since pretty soon you'll see their from address in their message.

Anyone who doesn't want you to know their address before sending a message is probably malacious and you don't need their email anyway. Anyone who doesn't mind will gladly give it to you when they ask you for your email address.