Faster Feeds Using FeedTree Peer-To-Peer

dsandler writes "Researchers at Rice University have just released version 0.7 of FeedTree, a peer-to-peer system for distributing Web feeds faster. Instead of polling feeds independently, FeedTree users cooperate to share news updates using multicast in Pastry, a scalable p2p overlay network. FeedTree reduces the update delay for existing RSS and Atom feeds to a few minutes without putting extra stress on the webserver (anyone who's ever been temporarily banned by Slashdot's RSS feed knows this is a real concern). Feed publishers can also choose to push digitally signed updates for immediate, tamper-proof delivery to subscribers. The client software (download) runs on Linux, OS X, and Windows, and works with any desktop feed reader."

Why?

WIth Bittorrent et al firmly established, why do we need another P2P?

Re:Why?

It's not "just another p2p", it's a p2p specifically for distributing newsfeeds. Bittorrent doesn't really work to well for that because it doesn't have the infrastructure and downloading the real feed would be easier than downloading the torrent first. This software bypasses any user interaction and grabs it off its established network.

Re:Why?

As already mentioned, this doesn't compete with Bittorrent, because bittorrent isn't designed for RSS feeds. Along with the file size issue idonthack mentioned (torrents are only a win when the size of the file being transferred is much much larger than the coordination overhead, generally not the case for RSS), BitTorrent is also not designed for files to change over time; it would require a complete overhaul of the protocol because the file hashes that are the foundation of the protocol would be constantly changing.

There is room for coordination with bittorrent, though; imagine a Pastry-based P2P feed that then used RSS enclosures to tie into a (trackerless?) BitTorrent feed for a fully distributed pod-/vid-/file-casting solution that anybody could run with no fear of the bandwidth involved.

Tack in some sort of P2P web system, and in theory, you could run a massively popular podcast/blog with millions of hits a day off of your cable modem. (Although something with a bit more upstreaming oomph would be good for the rarely-requested content that falls out of the P2P; anyhow, any ol' webhost could handle this kind of bandwidth.)

I think this is a worthy goal, as if nothing else, popular websites run for fun would no longer be faced with the dilemma of advertising to cover bandwidth costs or going offline.

They just reinvented netnews

It looks like they just re-invented the netnews protocol, which works in a very similar way.

Re:They just reinvented netnews

That's like saying IMAP just reinvented POP3.

This is designed for USERS to help each other get the very latest RSS feeds using p2p tech.

netnews is designed to let SERVERS help each other distribute messages posted by users.

I don't really see how it is a re-invention at all.

Feed Reader

What's the best OS X feed reader to use with FeedTree? I don't care for the way Safari handles RSS.

Does the client work on FreeBSD?

s/t - has anyone run this on FreeBSD? Perhaps it works with the Linux compat modules loaded? I'd like to try this out tonight, since I have 3 sites on my FreeBSD box that have feeds that are constantly being hit...this sounds like a solution for the long term.

Obligitory Simpsons quote.

MMMmmmmmm, Pastry.

Already getting hit by Shrook

I remember seeing something like this in my logs over a year ago. I would see lines like this in my access log:

66.177.198.139 - Anonymous [04/Apr/2005:03:04:17 -0500] "GET /rdf10_xml HTTP/1.1" 200 5322 "" "Shrook/76p (Distributed; +http://www.fondantfancies.com/shrook/distfaq.php [fondantfancies.com]) "

I haven't seen a hit from this in a while, perhaps that effort didn't gain much traction. Who knows if this one will... I never saw Shrook mentioned on Slashdot.

GMail RSS

I wonder: If GMail were to incorporate an RSS reader (the way Thunderbird does), it could potentially update many, many users with a single hit of each RSS site.

I'm leaning towards using RSS as a way to do announcements rather than maintain a mailing list. Rather than tell me you want me to send you updates (and deal with being potentially a spammer, deal with your unsubscribe, your email address change, etc.), just poll my site every so often (days, for the lists I'm talking about; hours, for Slashdot) and let it show up in your mail queue.

The idea isn't quite ready for prime time; too few people use RSS. But GMail could make that happen in one fell swoop. Well, two fell swoops: you'd need some sort of browser extension to make the little orange "RSS feed" button notify GMail.

I wonder if just having GMail (and hotmail, aol, and yahoo) handle that would solve the problem to the point where we no longer needed a P2P RSS distribution system.

Alternatively, if ISPs were to cache the RSS feeds the way some do with certain web pages, that might also take a lot of the load off. People will still impolitely set their RSS readers to check the feed every 10 seconds, but at least it never gets out onto the backbone if it's cached at the ISP.

Wow...

Try saying that headline 5 times fast!

Rice made Pastry, too.

As a Rice Computer Science student I would like to point out that Pastry [freepastry.org] actually originated at Rice, under Dan Sandler [rice.edu]. The first framework was in Java. You can see from his web page that he's responsible for FeedTree, too.

Microsoft Research became interested in the product and ported it to C#, effectively turning it into the form it is now. Many classes at Rice have now "backported" it, I guess you could say, and it's used for many of our classes that involve distributed networks, such as the current COMP 410 [rice.edu] class which has previously turned out distributed file and process system codename Voltron [rice.edu].

Here's a link to the paper [rice.edu] co-authored by Sandler and others at Rice.

License Terms?

I can't find any mention of the license terms on the Web site.

Does /. RSS push its updates to this?

An excellent project, it deserves to become dominant in internet
RSS news distribution.

It's nice to be able to browse the source code.

What can we do to encourage adoption of this, before some wretched
proprietary format tries to muscle in?

Web-Based News Reader

I personally use Bloglines [bloglines.com] - a web based news reader. This lets me check and read my subscriptions from home and work without having to read posts twice. Google Reader [google.com] is a similar application but has tagging and merges all your feeds into one.

The reason this is a issue

Is because RSS doesnt pay. There's no way to monetize the RSS-feed which often can be a large burden on a server in terms of CPU (if dynamic) and bandwidth.

Micropayments would solve this. Pay 0.001 for every reload automaticlly and you wouldn't need a solution like this. Fix that and solve thousands of small problems at once.

How does this relate to scribe?

Scribe [microsoft.com][Technical paper pdf warning!] is a framework to do very similar things. Is this an application developed on top of that? Scribe works by building a multicast tree of the participants too.
One interesting thing to note is that as a participant in scribe, you'll have to pass on notifications of feeds even if you're not interested in them, because you're a part of the tree and pretty much the only path to the guys below you. How does FeedTree deal with cheating/lying nodes that refuse to pass on messages? Also, to be a part of the overlay, you need to keep sending keep-alive messages. Not a big deal, I know, but I always thought Scribe was impractical for general use, but would work great for a restricted audience (like a large geographically distributed company) that can be "trusted".

A solution without a problem

I'm afraid I don't understand what problem this is solving. It's like a solution that's still looking for an problem to solve. As an end user, why should I care? I'm not trolling; I just don't get it.

Truly beneficial?

I've been thinking for quite some time of utilizing this type of P2P distributed caching proxy concept with many different protocols. RSS is just one possibility amongst many that could utilize the basic technology here. Some others might include distributed file systems, distributed caching http proxies, or even a Google competitor that uses a distributed P2P implementation of the database and utilizes everyone's everyday web activity to augment the spidering (i.e. every time anyone who is part of the P2P search network hits a site, a side effect is that they update the search index with the latest data from that site).

I'm not sure though that this is truly beneficial in terms of reducing the burden that RSS places on the Internet in general. Yes, it reduces the burden on the originating web site, but I believe it increases the total number of packets that must flow across some internet connection somewhere. So, it appears to be a mechanism for shifting the cost from one at the server to a larger total one at the clients, not a mechanism for helping the internet as a whole. I would in fact be positive that this is not beneficial overall except for the fact that it may have a beneficial reduction of the peak traffic on critical network backbones. But that would only be true if the overlayed network topology is either geographically optimized or is based on something that has an accidental relationship to geography.

New game in town...

The client software (download) runs on Linux, OS X, and Windows, and works with any desktop feed reader.

New game in town: never use the word Java. BTW, it doesn't run on Linux and Windows. Except if you install Java of course.

Why not use push?

I know it's a crazy suggestion, but instead of having hundreds of people polling a single RSS feed, why not have the server which hosts the RSS feed actually PUSH the updates out to the people who are interested?

We already have a nice and simple protocol (XMPP) which could be used for this, although admittedly PubSub isn't as final as it could be.

Similar in some ways to Miski (invented in 2000)

More details at How I Invented a Decentralised Scaleable Push-Based Micronews System in 2000 [1729.com].

If nothing else, my documented but unimplemented invention might be good prior art, should it be needed.

Pah!

Distributed peer-to-peer web 2.0 rss news updates? You young whippersnappers and your fancy-schmancy names!

In my day we simply called it gossip!

Re:So...

Now how do we get a plugin for something like this mass installed into browsers like that of flash, so that a web app like google reader (but independently developed, maybe specialized to other things) can take advantage of this idea?

That would make for a real nice way of creating awesome web based feed aggregation tools...

Re:So...

I wonder if we'll be sued for downloading copyrighted newsfeeds.

Re:Multicast?!?!?!1ONEONEONE

The Internet does not support IP multicast, so FeedTree uses application-level end-system multicast.