Using AI To Filter RSS Feeds

holden writes "According to a blog post, AideRSS has moved from closed to open beta. I've been using AideRSS over the past few weeks to filter my RSS feeds (including Slashdot and Reddit) and I've been quite impressed. They talk a bit about how the filtering system works, which apparently tracks a mixture of things, from pick-up in other blogs, to some clustering technology."

Some filtered RSS feeds

I'm not sure if it is bad form to comment on your own story, but here goes anyways :). You can take a look at the scored version of the slashdot RSS feed here [aiderss.com], or del.icio.us [aiderss.com] or my (holden's) blog [aiderss.com]. There is also a really cool widget I've put on the side of my blog which lets people subscribe to only posts of a certain quality (you can look at it here [holdenkarau.com]).

Title really sucks

It should read more like "AideRSS fianlly released" or "AideRSS goes live."

As for the article, what kind of person or group has too many RSS feeds to look through?
I'm asking because I really have no idea. I have linked the RSS bar in my Gmail to Tomshardware and Slashdot, but that's about all that I need....

Secret Sauce and GeoRSS

From TFA: ""Some of that data we show on the site itself: Technorati, del.icio.us, etc. Essentially, we're interested in measuring the 'social engagement' of each post. To make this a little less hand-wavy, I think we'll agree that a bookmark is nice but a comment involves more work, a trackback even more so, etc. - hence, engagement). Once we have all this data, we apply our 'secret sauce', which comes in a form of statistical analysis with respect to the author's previous history/posts. PostRank is not a global score, it's with respect to the blogger him/herself.""

Secret sauce? Why do I prefer open sauce? ;-)

One other way to filter RSS is by geographic location through using GeoRSS [slashdot.org]. However, the source RSS must be offered in GeoRSS for this geolocalization filtering to work... but it's only a matter of time, we'll get there. (hey, even slash has a plugin that works for publishing GeoRSS)

If Only ...

If only they could get the AI to do the work I'm missing out when I'm reading RSS feeds.

Download URL

Anyone know where you can download it?

Potentially scary side-effects already.

There are some companies out there(i.e. http://www.collectiveintellect.com/ [collectiveintellect.com]) that are using AI to mine RSS feeds and specifically the blogosphere, and selling that data to corporations for various reasons.

Lets say you're a drug company that is releasing a potentially controversial drug. You can mine the data of the blogosphere and issue press releases as a pre-emptive strike to larger media stories. This starts the real beginning of being able to effectively monitor and even potentially control some of the social aspects of the internet. I think it's a great innovation indeed, with potentially scary side-effects.

Personally it is nice to be able to filter through a billion RSS feeds to find information that I'm interested in though.

Another site using AI

Thoof [thoof.com] (disclaimer: its my website) uses Bayesian analysis (you could call it AI, so much as anything is AI) to determine what you are interested in reading, based on a variety of factors, including:

• The referring website (and what other people from that site liked)
• Your OS/Browser (and what other people with your OS/Browser liked)
• Your geographic location (and what other people close to you liked)
• What you yourself read

It also allows users to edit stories, a mechanism conceptually similar to a wiki, but with an additional voting process to help prevent abuse.

Unlike AideRSS, Thoof isn't an RSS aggregator, rather users submit stories, in a manner similar to Slashdot, Digg, and Reddit.

recursion

What if the 'other blogs' they 'pick up' on, are in turn using AideRSS to determine what to blog. The whole blogging thing really does seem like one giant feedback loop with only a few people generating actual useful content.

Artificial Intelligence on Slashdot

I suppose in these modern days when natural resources are being rapidly depleted by overpopulation and overconsumption, there had to come a time when we would start running out of intelligence... of course I wouldn't know because I'm a little short on it myself...
however
It is pleasing to see that scientists around the world have started to produce artificial intelligence to make up for the loss of natural intelligence, but I think that like everything else, perhaps it is also equally important that we conserve and recycle the little natural intelligence we have left and refine our methods to efficiently extract and use that intelligence to, uh, do something or other, but do it efficiently and without any needless waste. Yes, that's my point.

And to that end I see this Artificial Intelligence RSS Feed Filter as a great marvelous invention, because you see, it combines the old and the new, it uses artificial intelligence to extract natural intelligence efficiently and use it for something in a wonderful postmodern fashion. Now, modern invention assists the primitive natural.

Now, all we need is to have a massive SETI like project running this AI RSS Filler Feeder to search for signs of intelligence on slashdot. Oh, oh, cross my fingers, I hope my post makes it pass the filter...

Dupes!

Yes, but can it filter dupes?

Another half-baked open soucred approach

Way back in 2003 [srijith.net] I wrote some codes to do something similar. I called it Intelli-Aggie [srijith.net] and the code is released under GPL. It remains a developmental prototype as I got side-tracked.

IA works, as noted in the readme [srijith.net], by computing a relevance factor, which in turn is based on four other 'relv' - category relevance, feed relevance, keyword relevance and item relevance. I used it as my reader for quiet some time before moving over to 'better' readers.

Filtering RSS

I vaguely remember somebody saying the whole point of RSS is that you never get content you don't want because you have to subscribe to it in the first place. What's stopping us from unsubscribing instead of filtering?

Personalize instead

I think there are basically two kinds of RSS Feeds, either they show the latest news (last in first out) or they show an already sorted frontpage (e.g. "crowdsourced" like Digg); both are useful.

Using an AI to resort those feeds is definitely interesting from a coders point of view but trying to give some kind of objective view to a feed is probably not what the average user wants.

Why not do it the other way around and personalize them instead? Maybe it has been done before, but it would be nice if there was a reader to rerank (or even filter out) certain domains, keywords, tags and categories. It could take the given rank as the base score and then resort it according to the user's personal preference, e.g. if someone doesn't like politics he could give the keywords "Bush, Cheney, election, etc." a negative mulitplier and maybe the keyword "funny" gets a positive one. It could even consider the time of the day - politics in the morning and funny pictures during the lunchbreak or something.

Just a qick thought though, someone can perhaps come up with something better. Anyway, I am pretty sure that personalization is the better approach here.

Risks of non-algorithmic filtering

I view with alarm the increasing use of "artificial intelligence" to filter, screen, or otherwise judge human-generated material. In this case it's not enormously important, but it's part of a growing trend.

The issue is lack of responsibility or accountability, because at a certain level of complexity, it is no longer practical to understand or explain the basis of individual decision. The company can just say "the computer did it."

A few years back there was serious consideration being given to using neural nets or something like that to make judgements on loan applications. IIRC the proposed way of handling some sort of legal issues regarding accountability was to add to the system a subsystem that would automatically test the effect of hypothetical changes in the applicant's income. Thus the company could always say "this application was rejected because the applicant's income was too low, and would have been accepted if the applicant had earned X thousand more a year." Raising the question, of course, of whether this was the real reason. Or what it means to talk about "the real reason" in the case of a decision made by a neural net.

In the case of a neural net made of meat, it's possible to cross-examine the net and attempt to find out whether illegal bias played a role in the decision. In the case of an AI neural net, there may be bias built-in... but there's no way to ask the neural net itself about this, and unless the programmers did it deliberately and consciously and left a paper trail, it's pretty hard to find out about it.

Just what I need

Finally, I am one of those people who are swamped by news feeds. Some of the feeds I subscribe to are updated very regularly (the news ones) and I don't need to read everything that appears on them others (personal blogs) are infrequently updated and I want to read everything.

Two things I'd like to see:
An offline version; I know it's unlikely to appear (Web 2.0 business model and all that) but I'll never use the online one in the long term.

The ability to upload a bookmarks file filled with rss links. I don't want to have to manually upload all my rss feeds. Also it'd be nice to be able
to change the story levels for all of the feeds from the one page (radio buttons and a table?) rather than having to access each feed before setting the story level.

D.

My first thought?

What's Al Gore got to do with this besides inventing the Internet, and how can I get him to filter my RSS feeds?!

I would have replied to this earlier

But AideRSS filtered the post out...

Re:AI?

I guess that's your problem. Sorry, you don't get to define the terms.