Follow Slashdot blog updates by subscribing to our blog RSS feed


Forgot your password?
The Internet

Using AI To Filter RSS Feeds 53

holden writes "According to a blog post, AideRSS has moved from closed to open beta. I've been using AideRSS over the past few weeks to filter my RSS feeds (including Slashdot and Reddit) and I've been quite impressed. They talk a bit about how the filtering system works, which apparently tracks a mixture of things, from pick-up in other blogs, to some clustering technology."
This discussion has been archived. No new comments can be posted.

Using AI To Filter RSS Feeds

Comments Filter:
  • by holdenkarau ( 1130485 ) on Tuesday July 24, 2007 @10:12PM (#19978639) Homepage
    I'm not sure if it is bad form to comment on your own story, but here goes anyways :). You can take a look at the scored version of the slashdot RSS feed here [], or [] or my (holden's) blog []. There is also a really cool widget I've put on the side of my blog which lets people subscribe to only posts of a certain quality (you can look at it here []).
    • The secret sauce tastes like teen spirit.
    • "Using AI to Filter RSS feeds" only scored a 1.4 and is at the bottom of my page. Wouldn't you think those using an AI to filter feeds would be interested in a story about AIs filtering feeds? Seems like an automatic top of list.
  • It should read more like "AideRSS fianlly released" or "AideRSS goes live."

    As for the article, what kind of person or group has too many RSS feeds to look through?
    I'm asking because I really have no idea. I have linked the RSS bar in my Gmail to Tomshardware and Slashdot, but that's about all that I need....
    • Re: (Score:1, Funny)

      by Anonymous Coward
      I have too many RSS feeds to read through (but thats in part because I try and read the digg rss feed... ugh...)
    • I am probably an exception to the rule, but I just counted up the different individual news feeds in my NewsReader's (Default- RSS OWL [] Java OS) start-up OPML file, and there are 1074 unique feeds in it. Granted, there are a few major Mainstream News site feeds that I just recently updated the feed lists from their RSS pages, and haven't yet filtered out the many I consider to be irrelevant from them. Two I recently rescraped but haven't filtered are are McClatchy News and the NYTimes, but even after filter

  • From TFA: ""Some of that data we show on the site itself: Technorati,, etc. Essentially, we're interested in measuring the 'social engagement' of each post. To make this a little less hand-wavy, I think we'll agree that a bookmark is nice but a comment involves more work, a trackback even more so, etc. - hence, engagement). Once we have all this data, we apply our 'secret sauce', which comes in a form of statistical analysis with respect to the author's previous history/posts. PostRank is not a global score, it's with respect to the blogger him/herself.""

    Secret sauce? Why do I prefer open sauce? ;-)

    One other way to filter RSS is by geographic location through using GeoRSS []. However, the source RSS must be offered in GeoRSS for this geolocalization filtering to work... but it's only a matter of time, we'll get there. (hey, even slash has a plugin that works for publishing GeoRSS)
    • Re: (Score:3, Informative)

      by bergie ( 29834 )

      Geolocation is a possible additional filter (think "local news" section of a newspaper), but I guess most people are interested in items from their field of interest regardless of the physical location where the post was made.

      I made some experiments [] on a more open source version of the "secret sauce". It seems quite easy to determine relevance of posts using the various social news services out there.

      • Actually you'd be surprised. What about the economies of scale in running a centralized
        event calendar, with the advantages of letting the reader select it's own idea of "local"?
        (Obviously you'd need to apply some minimal pre-filtering for "local" on the server)
  • If Only ... (Score:5, Funny)

    by nxtr ( 813179 ) on Tuesday July 24, 2007 @10:52PM (#19978935)
    If only they could get the AI to do the work I'm missing out when I'm reading RSS feeds.
  • Anyone know where you can download it?
    • Re: (Score:3, Interesting)

      Someone can correct me if I am wrong but it looks like your supposed to put your feeds into the website then link to the one feed there.

      Seeing as half my feeds are internal work related and the fact I don't want someone profiling all feeds I am reading I won't be using the service.
      • Someone can correct me if I am wrong but it looks like your supposed to put your feeds into the website then link to the one feed there.
        yes, "you put your feed in there", exactly.

        the privacy concerns for this seem to mirror those of any of the public feed browsers like Google Reader, it's probably a bad idea use them for private feeds
  • by gethoht ( 757871 ) on Tuesday July 24, 2007 @10:59PM (#19979019)
    There are some companies out there(i.e. []) that are using AI to mine RSS feeds and specifically the blogosphere, and selling that data to corporations for various reasons.

    Lets say you're a drug company that is releasing a potentially controversial drug. You can mine the data of the blogosphere and issue press releases as a pre-emptive strike to larger media stories. This starts the real beginning of being able to effectively monitor and even potentially control some of the social aspects of the internet. I think it's a great innovation indeed, with potentially scary side-effects.

    Personally it is nice to be able to filter through a billion RSS feeds to find information that I'm interested in though.
    • Re: (Score:3, Insightful)

      by rm999 ( 775449 )
      "This starts the real beginning of being able to effectively monitor and even potentially control some of the social aspects of the internet"

      I fail to see your reasoning. Companies have always been able to "monitor" blogs and subscribe to RSS feeds. And they aren't controlling the social aspects of the internet at all. A press release has always been a standard communication means of corporations; as long as they aren't creating fake blogs, I don't think they are trying to control any aspect of the social i
    • I see what you are saying. In the past, powerful corporations might say, "We cannot do actions that might put us in a poor light, because public opinion would turn against us." Now, they can instead measure public opinion closely and watch web-trends based on RSS keywords. They can measure their actions by the result of the "blogosphere" and then gauge their next nefarious action accordingly.

      In reality, however, I think the common practice of spider-intelligence-gathering is simply another tool for marke
    • Re: (Score:3, Interesting)

      by dada21 ( 163177 )
      There are some companies out there(i.e. []) that are using AI to mine RSS feeds and specifically the blogosphere, and selling that data to corporations for various reasons.

      Sounds like a fantastic market, actually. I recently picked up a client in the casino management market because I had made some comments on a blog regarding their lack of insight towards proper marketing and keeping a decent percentage of return customers. They actually contacted me, and I've spent a lar
      • A bit tangential but lately in server logs I have access to I've noticed a proliferation of 'vertical search engine' bots, which do not claim to ever be planning to provide the data they acquire at the websites' bandwidth cost in any manner which could possibly be deemed as reciprocal. Even more troubling are a few sites that throw mad wget type bots at sites, with user strings claiming to be a common browser, without concern for bandwidth spikes by using decent time intervals between GET requests, and wil

  • by Sanity ( 1431 ) * on Tuesday July 24, 2007 @11:06PM (#19979071) Homepage Journal
    Thoof [] (disclaimer: its my website) uses Bayesian analysis (you could call it AI, so much as anything is AI) to determine what you are interested in reading, based on a variety of factors, including:
    • The referring website (and what other people from that site liked)
    • Your OS/Browser (and what other people with your OS/Browser liked)
    • Your geographic location (and what other people close to you liked)
    • What you yourself read
    It also allows users to edit stories, a mechanism conceptually similar to a wiki, but with an additional voting process to help prevent abuse.

    Unlike AideRSS, Thoof isn't an RSS aggregator, rather users submit stories, in a manner similar to Slashdot, Digg, and Reddit.

    • n/ [] []
      What I find interesting is, it is one of the verrry rare examples of 'internet 2' service that you can own yourself (instead of registering here or there for more ads or worse).
      A downside of Sux0r is it seems not having evolved for a couple of years (but still works, possibly that's why :-)
      I for one am desperately waiting for a *local* RSS agregator which would allow *me* (and not some site's AI) to Bayes-filter my se
    • by jamshid ( 140925 )
      Wikipedia says "intelligence is a property of mind that encompasses many related abilities, such as the capacities to reason, plan, solve problems, think abstractly, comprehend ideas and language, and learn."

      Fancy stats != intelligence
      • by Sanity ( 1431 ) *

        Fancy stats != intelligence
        Are you sure that intelligence isn't simply the combination of billions of neurons each processing information?

        I have a bachelors degree in Artificial Intelligence, and I certainly wouldn't claim that intelligence doesn't simply boil down to mathematical computation at some point, indeed, I suspect that it probably does. Its just that we don't understand it yet.

  • recursion (Score:5, Insightful)

    by shird ( 566377 ) on Tuesday July 24, 2007 @11:22PM (#19979147) Homepage Journal
    What if the 'other blogs' they 'pick up' on, are in turn using AideRSS to determine what to blog. The whole blogging thing really does seem like one giant feedback loop with only a few people generating actual useful content.
  • by Anonymous Coward
    I suppose in these modern days when natural resources are being rapidly depleted by overpopulation and overconsumption, there had to come a time when we would start running out of intelligence... of course I wouldn't know because I'm a little short on it myself...
    It is pleasing to see that scientists around the world have started to produce artificial intelligence to make up for the loss of natural intelligence, but I think that like everything else, perhaps it is also equally important that we conse
  • Yes, but can it filter dupes?
  • Way back in 2003 [] I wrote some codes to do something similar. I called it Intelli-Aggie [] and the code is released under GPL. It remains a developmental prototype as I got side-tracked.

    IA works, as noted in the readme [], by computing a relevance factor, which in turn is based on four other 'relv' - category relevance, feed relevance, keyword relevance and item relevance. I used it as my reader for quiet some time before moving over to 'better' readers.
  • I vaguely remember somebody saying the whole point of RSS is that you never get content you don't want because you have to subscribe to it in the first place. What's stopping us from unsubscribing instead of filtering?
  • by Catil ( 1063380 ) on Wednesday July 25, 2007 @05:20AM (#19980803)
    I think there are basically two kinds of RSS Feeds, either they show the latest news (last in first out) or they show an already sorted frontpage (e.g. "crowdsourced" like Digg); both are useful.

    Using an AI to resort those feeds is definitely interesting from a coders point of view but trying to give some kind of objective view to a feed is probably not what the average user wants.

    Why not do it the other way around and personalize them instead? Maybe it has been done before, but it would be nice if there was a reader to rerank (or even filter out) certain domains, keywords, tags and categories. It could take the given rank as the base score and then resort it according to the user's personal preference, e.g. if someone doesn't like politics he could give the keywords "Bush, Cheney, election, etc." a negative mulitplier and maybe the keyword "funny" gets a positive one. It could even consider the time of the day - politics in the morning and funny pictures during the lunchbreak or something.

    Just a qick thought though, someone can perhaps come up with something better. Anyway, I am pretty sure that personalization is the better approach here.
    • Re: (Score:2, Insightful)

      spot on.

      i also think their should just be a thumbs up/ thumbs down option which would save you typing in.
  • I view with alarm the increasing use of "artificial intelligence" to filter, screen, or otherwise judge human-generated material. In this case it's not enormously important, but it's part of a growing trend.

    The issue is lack of responsibility or accountability, because at a certain level of complexity, it is no longer practical to understand or explain the basis of individual decision. The company can just say "the computer did it."

    A few years back there was serious consideration being given to using neural
    • Thus the company could always say "this application was rejected because the applicant's income was too low, and would have been accepted if the applicant had earned X thousand more a year." Raising the question, of course, of whether this was the real reason. Or what it means to talk about "the real reason" in the case of a decision made by a neural net.

      How is it any different from dealing with a person? You have no idea if what somebody tells you is "real." And people can get hunches, where they feel

  • Finally, I am one of those people who are swamped by news feeds. Some of the feeds I subscribe to are updated very regularly (the news ones) and I don't need to read everything that appears on them others (personal blogs) are infrequently updated and I want to read everything.

    Two things I'd like to see:
    An offline version; I know it's unlikely to appear (Web 2.0 business model and all that) but I'll never use the online one in the long term.

    The ability to upload a bookmarks file filled with rss links. I don'
  • What's Al Gore got to do with this besides inventing the Internet, and how can I get him to filter my RSS feeds?!
  • But AideRSS filtered the post out...

Forty two.