RSS & BT Together?

AntiPasto writes "According to this Yahoo! News article, RSS and BitTorrent could be set to join in a best-of-both-worlds content management system for the net. Possible?" Update: 03/17 21:39 GMT by T : Thanks to Steve Gillmor, here's the original story on eWeek to replace the now-dead Yahoo! link.

Re:If he wants to save bandwidth.

[djh101010]

A base Akamai contract starts at $2,000 a month for a 1Mb/second bandwidth allowance. Not sure if many/any Open Source projects have a budget for such.

Akamai is great for offloading bandwidth and speeding up customer's page load times, but you're paying for the bandwidth one way or another.

Re:I'd rather see BitTorrent improved in more...

[Anonymous Coward]

setting up a tracker isn't hard.

create .torrent files
put .torrent files into a folder.
run bttracker.py telling it which torrents are allowed (the location of that folder) and a folder to use for tmp files.

run btdownload.py like you normally would when resuming a download.

send the .torrent file to your friends, random websites, post a link to in on /., send an email with it to **AA (attach a goatse/tubgirl pic with that last one too)

(for more details RTFM! or STFW!)

Konspire2b

[Dooferlad]

Konspire2b [sourceforge.net] looks like a better option than BitTorrent for distributing news. You could have a channel mapping to an RSS feed and just wait for the news to come to you. No polling intervals and low bandwidth requirements for the operator. With BitTorrent you still have to poll for updates and this removes that requirement.

Re:RSS polling intervals

[costas]

The real problem isn't the polling intervals, is that most RSS readers/spiders do not respect HTTP 304 (Not Modified). RSS is ideal for Etag/Not-Modified-Since behavior, but no, most spiders are still too lazy to implement this.

My newsbot (in my .sig) creates dynamic RSS feeds, customized for each agent; I thought that was a great feature to give users, but it's getting overused by some spiders hitting the site every 15-20 minutes, w/o listening for 304s...

Re:RSS polling intervals

[Darren Winsper]

Considering the official bittorrent client has the --max_upload_rate option, it's not much of a hack. I normally set it to around 15K/sec, to prevent it flooding my upload and making ping times bad for my housemates.

Re:RSS polling intervals

[passthecrackpipe]

The Slashdot polling timer is broken - I feed every 61 minutes, and still get kicked out one every week or so. I appreciate that they want to keep their b/w as low as possible, but for what pretends to be a news site, you have to let people be up to date. maybe a nice subscriber option (hint hint)

Re:RSS polling intervals

[bongoras]

1, BT lets you throttle your upload now. 2, if you do it, your download is also throttled. 3, if you want to modify btdownload.py so that it lies about how much it's uploading in an effort to get faster downloads, have fun. It won't help you because BT itself doesn't trust what the client says, it still sends only as fast as it's getting.

Re:BitTorrent is no-go for small files..

[SheepHead]

I thought he was talking about distributing BitTorrent links through RSS rather than sending each RSS news reader the full content of the page with graphics, etc.

So you send out a new torrent through RSS referencing your new page instead of the regular RSS content, and your viewers use BitTorrent to work together to get the content from you without putting all the strain on your server. A .torrent file would be a lot smaller than a full RSS feed with images like he was using in the example.

Makes more sense that way.

Whoah. </keanu>

[CrystalFalcon]

This is the first time I've heard FidoNet mentioned in... must be almost a decade. It's like the huge amateur network (which for a brief period outnumbered the Internet in raw node count, mind you) never existed.

Anyway, FidoNet was not without its share of problems. The killing bullet, I'd say today, was the social factor - there were too conservative forces clinging to backwards compatibility at the cost of anything. Anything had to work with the most basic piece of software; this effectively shot progress and evolution dead.

Not that there weren't attempts. There were. They just weren't successful.

Anyway, setting up echoes would have the same problems as FidoNet echoes. The number one problem was typical for Slashdot: DUPES!

Echoes were set up so that one node relayed a message in an echomail forum to its other connected nodes for a particular echo, effectively creating a star topology, different for each forum. However, since each sysop just wanted the echo linked, he would just hook up to somewhere, and forget about it. Then, others would hook up from him, and all of a sudden somebody had hooked up to two different valid uplinks.

The result? The star topology all of a sudden had a loop in it. Messages would keep circling (since FidoNet used dedicated dialup lines, latency between nodes was typically in the hours range) and dupe filters were created.

All of those filters and filter-enabling tags were optional, of course. After all, you couldn't mandate an operational node to change its behavior, you could just ask nicely.

Political play to no ends. :-/

Anyway, there were many other funny effects with EchoMail. Crosslinking was another - when one echo got linked to another at a node, so that all messages in echo X would enter echo Y at that node and vice versa. The most exotic of these was when a religious echo got crosslinked with a fantasy humor one -- through crosslinked physical directories at a node (the FAT pointers for the different directories hosting the two echoes pointed to the same location on the disk). Anyway, much hilarious discussion ensued, and not many understood much what people were trying to say in the crosslinked echo. :-)

/ former sysop and NEC in FidoNet

Use conditional GET, not HEAD

[NonaMyous]

An even better behaved program will use conditional GET instead of HEAD. For more info, see HTTP Conditional Get for RSS Hackers [pastiche.org] :

The people who invented HTTP came up with something even better. HTTP allows you to say to a server in a single query: "If this document has changed since I last looked at it, give me the new version. If it hasn't just tell me it hasn't changed and give me nothing." This mechanism is called "Conditional GET", and it would reduce 90% of those significant 24,000 byte queries into really trivial 200 byte queries.

Re:I'd rather see BitTorrent improved in more...

[ikewillis]

I've taken a throw the baby out with the bathwater solution and have implemented BitTorrent-like download swarming with a server that stores a heirarchical filesystem and transfers that are highly server regimented:

http://pdtp.org/ [pdtp.org]

Re:RSS polling intervals

[Dave Beta]

So explain how I get 230K/s download with only 10K/s upload using the ABC client?

That isn't unusual. If there are plenty of uploaders with plenty of upstream capacity, you can expect fast downloads pretty much without uploading anything. The BitTorrent idea really comes into play when everyone is trying to download at the same time. It guarantees some level of fairness since other clients will give you faster downloads if you are being generous in uploading to them.

Strange things do still happen though. Sometimes my download rate will actuially increase when I throttle back on my uploading. I'm not sure what the other peers are thinking when they let that happen!