RSS And BitTorrent, Together At Last

eoyount writes "Wired has an interesting story about a really simple idea I wish I had thought of. Transferring large files across the Internet isn't easy for your average joe, but a combination of RSS and BitTorrent technology might just make it easier - Slashdot ran a previous story on the theoretical blending last year." (LegalTorrents is run by the strangely familiar simoniker, who wrote a short piece on the O'Reilly Network about how it was set up, and offers observations on how well the combination fares.) Update: 03/17 21:45 GMT by T : Ernest Miller submits two related postings he's written on RSS+BitTorrent, a combination he calls "broadcatching."

RSS + BitTorrent = Broadcatching

I've been writing extensively on Corante about RSS + BitTorrent, which I call "Broadcatching" here: Broadcatching Archives [corante.com] See, for example, RSS + BitTorrent Roundup - Broadcatching Isn't MS Active Channels [corante.com] and First Broadcatching App Available! (And Related News) [corante.com].

Hack your TiVo for fansubs

The way I figure it, with this bittorrent-RSS combination and a slight modification of torrent watching sites like animesuki [animesuki.com] we will essentially have a fansubbed anime online tivo at our disposal. Actually, you could have probably done that even without RSS, though it does simplify matters. The only limitations are our bandwidth and hard drives. Which actually are pretty limiting these days, especially with p2p being frequently capped.

Hell, you could modify an actual TiVo with broadband for exactly this sort of thing, and it needn't be limited solely to anime either. I'm sure it'll be popular with overseas watchers of American TV as well.

The international media and internet companies need to face facts and realize that Video On Demand is a reality and is already extremely popular - but that the shows people are demanding are not the ones the companies have been providing through their own limited, misfocused, and (most importantly) redundant services. Until we see simultaneous worldwide release of all media (including DVDs released simultaneously with the theatrical release) they will find themselves losing what should have been their easiest sales - those to impatiently eager fans.

Re:Hack your TiVo for fansubs

Large amounts?

Can someone is can make subtitles near to or exceeding professional quality ones for free? Translating, editing, timing, and typesetting? They can and they do. That is why fansubs exist. Do they bother with sound? No, because dubs are very difficult to get even close to comparable with the original language, if at all, but subs are easy and require only a little quality control. In fact, the original producers would be wise to cultivate and sponsor these international volunteers to do the the translations and editing for them (because timing and typesetting are the most technical parts but can be applied to all the translations). Plus, unlike sound subtitles require a negligible amount of bytes compared to the video.

Can someone distribute content for essentially no cost to the producer? Perhaps you ought to read the article again. Obviously the answer is yes. This is why digital fansubs are far more widespread and popular than the old VHS variety ever was. This is why the parent of this thread was referring to articles that describe how indy bands and movies can make themselves known and spread their work.

Are international menus hard/expensive to make? No, unless you make it difficult for yourself in the first place. Frankly, I'm really only interested in watching the show, and a lot of the overly flashy and slow menus out there only make things annoying. Do you even need menus for online distribution? No, since generally it's just a single movie/song/album/file. How about packaging? No.

What about when electronic distribution is not available, i.e. poor countries? Well, those guys on the blankets on the sidewalks seem to be able to manage. Certainly I've seen a few bootlegs from Hong Kong in my time with laugably bad english but probably decent chinese, and they manage. In both cases they seem to be catering to people who are priced completely out of the legitimate market rather than simply unable to access it due to a lack of translation. Which is where black markets have always taken over.

If file transfer is hard for the average Joe...

...combining RSS and torrents is not going to solve the problem. This is the most complicated solution to a non-problem that I've seen since someone paid me to design something.

OK, newbie question

where can I find clear info regarding what is RSS exactly ? Isn't it somehow like what microsoft tried to do a couple years ago with their "Active Desktop" (c) TM concept ? Or am I completely way off ?

Re:OK, newbie question

RSS feeds are an easy way to move news from site to site. For example, here is Slashdot's RSS feed [slashdot.org]

You can find more information here [harvard.edu]

--
Real-time deal updates, over 400 a day! [dealsites.net]

Re:OK, newbie question

RSS is "Really Simple Syndication" and it's best thought of as a spinoff to XML. It's a language under which blog-type news-channels can publish their content using, and then the user can use an RSS client that can group stories together into whatever sequence the user wants to see.

It's also seen as a effective way to replace e-mail mailing lists. Instead of getting your newsletters in your e-mail client, open them up in your RSS client which works on a pull basis rather than a push basis, but can still present the content to the user just like an e-mail program might.

It's very different than Active Desktop... that was just the idea of letting IE browser windows be part of the Windows Desktop level so that users could have a frequently-refreshed mini-page of content on their desktop.

Neato

Now, how do we aim that at the web in general (and automagically) to avoid the slashdot effect?

A new browser protocol? Aim your browser at

bthttp://www.victim.com

and let it rip?

Re:Neato

No new browser protocol is needed.

The technology is already available at http://freecache.org/ [from the peeps @ archive.org]

I don't why many others have jumped on the bandwagon yet.

Re:Neato

Wow! You're right, with one smallish exception:

Please note that you cannot submit a whole site to FreeCache as in http://freecache.org/http://www.rocklobsters.com/ This will not work as only index.html will be cached. You have to prefix every item that you want to have cached seperately.

Using the last THG article as an example, either the Slashdot story would need to point to each page individually via freecache redirection or Tom's Hardware would need to do it.

Not quite as transparent as incorporating BitTorrent into the browser.

Although it's cool...

Although it's cool that companies are finding legit uses for BT (I believe the Worlds of Warcraft beta is being distributed this way), I'm not sure the legal departments are up to speed yet. To quote one of the fellows in my IRC chat:

"Hrm, WoW is bing distributed by Bittorrent. Meanwhile, I get angry phonecalls from Vivendi to shut down Bittorrent."

Yay for technical advances, but can commercial interests fully embrace it without killing the "evils" of it?

The problem with bittorrent

The problem with bittorrent is that a lot of users disconnect as soon as their download is finished. Won't this be an even bigger problem with game downloads (specifically multiplayer games) since even if the users knows they should stay connected afterwards, they might not since it would lag their game?

Arguably, yes...

...but in that case, you're no worse off than before. And realistically, if there's some huge download I'll usually just start it before I head to bed. Of course, if you're sitting there counting down the seconds until it's done, that's different...

Kjella

Re:The problem with bittorrent

No, because if someone downloading a 100MB patch uploads even 10 MB during their BT session and closes it right after it's downloaded, that's still 10 MB the main servers don't have to send. Take Blizzard for example. Right now, their company's servers have to send 100% of the patch files. With BT, if they can cut that to 50% or even 80%, that's a huge benefit.

When you don't think of it in terms of people uploading movie files, and think in terms of companies using the technology to ease load on their web servers, now you're looking at BT the way the author intended!

Not everyone can contribute

Since data sent equals data recieved within a BT swarm, and some people will act as seeders and continue to send more data than they recieved, you will always have people who will simply not have the opportunity to contribute to the swarm, mostly at the tail end. And of course many folks have their uploads limited or even completely cut off.

The real problem with bittorrent is that by enabling efficient transfer of large files, people are transferring larger files. And the service providers simply do not have the capacity for everyone to be sending those large files. They may advertise unlimited access but kids they really aren't set up for it. To say nothing of the fact that the way the internet is structured now is no longer geared towards everyone being as able to send as well as they are to recieve.

Really, the internet and its billing structure should be geared towards billing by amount received, and not amount served, and widespread implementation of load-sharing protocols like bittorrent. It would be far more efficient and fair, and would encourage people to limit their consumption rather than penalizing inadvertently popular unsupported sites.

Re:The problem with bittorrent

That's okay, if only one person is downloading, then that person will get it from original source, right? I believe this is how it works.

If the download is not popular, than the orginator of the content can handle the bandwidth. Bittorrent's benefits kick in when something is popular, where there are simultaneous downloads at any given moment. If demand trickles back to one request every hour, than obviously the originator can handle it. Once it is no longer relevant, the orginator of the content can disable the tracker.

Bittorrent is a p2p network that works BETTER the more people are using it. Once everyone disconnects, then you revert to the worst case scenario, which is just straight downloading.

So don't worry, disconnecting after you finish is okay. You did your civic duty by sharing the bits while your download was in process. Enjoy your game guilt free.

I believe this is how bittorrent works. If I'm wrong, I'm sure I'll be corrected within - 3 - 2 -1 NOW

Re:The problem with bittorrent

So don't worry, disconnecting after you finish is okay. You did your civic duty by sharing the bits while your download was in process. Enjoy your game guilt free.

Maybe...although it is much nicer if you let your u/l to d/l ratio is at least 50% - 100% is even nicer. You can can quit right after you finish, but it would be like only sharing 10% or so of your files on P2P networks that you downloaded. A step up from a true leech but not sharing completely.

Re:Why not do a non-linear download?

BT is non-linear as you suggest. The n'th person gets the n'th chunk. This still allows for (randomly-caused) relative scarcity of certain chunks (although they are not the last ones!), and that is the problem you notice up around 98%.

No, seriously, try playing a partially complete BT download of an AVI with a player that doesn't look for the index (mplayer, DivX, etc.). The file is missing random chunks, not the end.

Uphill battle?

Many ISPs and college campuses block P2P ports, BitTorrent included. I'm not sure that 'news' is a compelling enough reason to have many (or any) of them change their policies.

SuprNova.org ?

This sounds like exactly what SuprNova.org needs. It would relieve some of the server load on their main pages and would enable them to serve more .torrent files.

-molo

Re:SuprNova.org ?

You can already subscribe to SuprNova feeds right here. [varchars.com] Half way down.

Re:SuprNova.org ?

Suprnova is pretty much the major culprit in associating Bittorrent with illegal file trading. Full movies, warez, etc make up the bulk of their content. It's making things difficult for legit uses of Bittorrent, of there are many. Beware if you download anything off Suprnova, the RIAA and MPAA are watching those torrents and gathering some nice logs of IP addresses and times. Remember that by participating in a Bittorrent you are uploading.

Too many good uses for Bittorrent to let the warez kiddies spoil it for us.

Bah.

People keep trying to make BitTorrent something it isn't. And really, we should be fighting its corporate adoption in any form, as it's simply an attempt to shift server bandwidth costs to the client. ISPs eat that right now, but we're going to metered access if this keeps up.

Which is effectively getting us to pay for website access/services, but instead of giving the money to the content creators we'll be giving it to ISPs instead and paying in bandwidth besides. So this is a bad idea.

Re:Bah.

You seem to believe that 'corporations' should pay whatever it takes to upgrade their servers and bandwidth in order to give you decent download times for free. Heh. If BT lets me get fast download times at the cost of using some of my mostly idle upload bandwidth, I think its a great idea.

As for ISPs metering bandwidth, guess what, you have to pay for what you use anyway, otherwise the ISP doesn't stay in business. It doesn't matter whether its metered or a fixed $30 or $60 / month. It has to cover their costs. If you're complaining that your cost would go up with metering, its because you think that you use a lot more bandwidth then everyone else. So you're just trying to shift the costs to the people that don't use as much. Pot, meet kettle.

Tinfoil Hat

Sounds like an OK idea, but is it just me or does anyone else think that there is just a bit too much hype in the *media* about this. They don't usually pick up on good ideas and try to make them critical mass and the "next big thing".

Syndication is a great idea, I like RSS, (does BitTorrent even work under Linux?)--but why on earth all the *orchestrated* hype?

Enough to make me try Freenet again. Harrumph

Re:Tinfoil Hat

does BitTorrent even work under Linux?

Uh, yes...

Here [debian.org], here [rpmfind.net], and here [sourceforge.net].

Covered this previously just before christmas

In this [slashdot.org] Slashdot article, Yahoo reported that things might be starting to come together. Looks like it's happened!

However, I'm a little concerned - BitTorrent has a lot of initial overhead (setting up trackers, and all the protocol stuff). I'm not sure if it would be wise for small files?

More ways for crap to flood in.

This sounds like a wonderful melding of two current technologies...
However, remember when cable gained enough steam to warrant not one but many 24hr cable news networks? We are now blessed with an overabundance of crappy sensationalist "reporting". I do NOT want cnn/msnbc/fuxnews/etc. landing on my HD.

If an individual set up a feed for say, a favorite game or movie alone, I would subscribe. But most webpages I read, I gloss over quickly then am done with.
If I, and everyone else had subscribtions to all of the media content of their favorite websites delivered autonomously, the majority of it getting thrown out quickly...

think of the bandwith, the poor helpless bandwith, won't somebody please think of the child., er bandwidth!?

A question for slashdot

Is bittorrent open to the same legal prying as Kazaa/fasttrack and gnutella by the RIAA? I would assume so. Just curious.

Good, but not perfect... yet.

BitTorrent's weak spot has always been thedistribution of the torrent files in the first place. If there isn't a torrent file on the conent provider's page, where do you look?

RSS+BitTorrent, is a step closer to a better web. It almost answers the problem of pointing your client at an actively downloaded torrent by steering users twoard a slimmer and more flexible protocol.

IMO, maybe some kind of 'standard' torrent directory/lookup that is guarnteed to be traded by all torrent clients is the right ticket; kind of like a DNS for media. The RSS+Torrent scheme is good, but all it does is displace the complexity of the matter onto a new protocol and rely on everyone hitting the same feed to begin (the problem Torrent is trying to eliminate).

It does however, make it easy to make distributing torrents a lot more dynamic. Neat stuff.

Already done - Konspire2b

Isn't this just Konspire2b? Konspire2b was designed specifically for this purpose:

Konspire2b [sourceforge.net]

Essentially you subscribe to channels which push content instead of pulling.

Compared to Bittorrent [sourceforge.net]

This is an exhaustive analysis (with pretty charts) why under the above scenario (pushing content, as opposed to pulling), Konspire2b is much more efficient.

Re:Already done - Konspire2b

Unfortunately, one assumption at the beginning, that the cascading model is best-case performance for BitTorrent, is completely wrong. It's actually worst-case performance.

A scattered model gives BT as taking O(log(number of people)/(number of chunks) + 1) time for everyone to download the whole file instead of O(sqrt(number people)) as claimed in the article.

Had an Idea like this a long time back...

But i never have time to implement.

I wanted on demand television that you could find from mirc downloads and then eventually BitTorrent. The idea would be for a really nice multimedia center attached to your TV that would download shows that you missed or if you couldn't record it (conflicts). Updates for popular programs could be downloaded and installed when the user attempts to update (as opposed to a live update). Harddrive sizes are definitely big enough to handle. The only challenge in my last implementation (which includes HTTP, MiRC XDCC, FTP downloads) was the average joe creating Torrent files so they could share their favourites with their friends. There would need to be some sort of authentication security to prevent everyone from downloading as well.

How long before ...

1. spammers learn how to tap into channels to use everybody else's bandwidth to deliver ads as content
2. it grinds to a halt from massive amounts of pr0n, warez, divx;), mp3 music, etc.
3. it is used to send virii to people, eroding trust of it

???

Sorry if I seem like I'm trolling but these questions will be asked at some point

Re:How long before ...

I think the problems you ask about are all problems inherent to a push medium (e.g. email) -- which AFAIK RSS is not. That is, the end user decides which RSS feeds to poll, and when to poll them. Therefore, if an RSS feed starts sending viruses or otherwise being malevolent, people will just stop using it and move on to other RSS feeds.

(Someone correct me if I'm wrong about this)

What a great idea!

The best thing about this idea is that it plays right into the strength of bittorrent - namely, having a large number of people trying to get the same content at the same time. Since everyone will get the RSS feed at roughly the same time, there will be a large number of peers to share the load for bittorrent.

The funny thing is, I ran into Andrew the other day, and he was just gushing about this new idea he had! I had no idea what he was talking about at the time. Guess I missed my chance to post a story on slashdot.

we'll use this on ampfea.org

right now we've got a 90's-style ol' skool web interface for distributing our media, teamed up with ol' skool ftp/http mirrors distributing things around the globe.

but, we'll definitely use an rss-fronted bittorrent network, if and when it can actually be smoothly integrated with our existing setup.

ampfea stands for 'a meeting place for electronic artists' [ampfea.org] and its a community-supported media hosting/community service for a bunch of muso's ... we are prime users for free, open, public, easy-to-use media sharing technologies, and if the big-guns aren't using it, we share are happy to!

check out some of our files sometime. its all home-made music... [ampfea.org]

Anime

This site [mircx.com] has been offering RSS feed of links to BitTorrent animes for couple of months. Firefox and RSS reader extension are great for pulling down all BitTorrent links.

Almost Nirvana

One step closer to Nirvana. Bittorrent + RSS + gentoo emerge --update world. Or how about something that uses software-suspend to automagically hot-swap in the latest bleeding edge kernel? Maybe the Hurd allows on-the-fly kernel upgrades.

RSS's "pull"/polling model is the real problem

Distribution of RSS isn't so much a problem as is the fact that web servers are hammered by RSS connections as RSS syndicators must continue polling the web server to identify when the RSS document updates.

This problem is easily addressed with multicasting. All a server need do is send a multicast datagram to notify all RSS syndicators that the RSS document has been updated, at which time the syndicators can fetch the new document.

Multicasting

True multicast could help, too. But seeing as cable companies cleverly bought the routers you use (unless you're in academia or the military or both), they're just NOT going to turn on multicast routing any time soon, unless more people are aware of the possibilities and start leaning on them HARD.

P2P: The Meme War Continues

In the left hand corner: BitTorrent+RSS is good for News

In the right hand corner: Hackers Embrace P2P [yahoo.com]

This article is more Wired garbage.

"I could wake up in the morning and find the latest recordings from my favorite band loaded into my portable MP3 player, and just pick it up and go."

I don't understand where they are coming from here. If I am going to pay to download music, which consists of relatively small files, I am not going to run a BT to help out an online music store.

When they start mentioning uses so far off the base of reality, the whole article starts to smell of BS. Especially since the slowest part of the MP3 experience tends to be copying music from the PC to the player.

Typically, I load new tracks on my ipod before leaving in the morning. I'll tag the stuff, then transfer it before I hop into the shower. As far as downloading goes, I can download a whole CD of music in ~10 minutes. The only way the article's method would be worth doing would be if you invested in huge libraries of online music purchases daily.

On that note: Please quit looking to solve problems that don't exist.

Re:This article is more Wired garbage.

Hey, quit bitching and open your mind a little bit.

Let's say your favorite band just went on tour and as part of a promotion they decided to post a few songs and videos "bootlegged" from each concert.

Now, it might get kinda anoying to load up their page every couple of days and click on each link to download the media. However, they could post an RSS feed with BitTorrent links that you subscribe to just once. Everytime a new bootleg goes online, the RSS feed gets updated, and the content gets downloaded to your computer automatically.

Where would we be if everytime the Internet was mentioned 50 years ago, people ranted and raved about how the postal service already solved the problem of distributing content?

This is seriously cool stuff, you are just too closed minded to realise it.

is this the same thing as konspire2b (kast)?

A million years ago (1998?) Wired published a whole edition on Push as the Next Big Thing. It was the first time I was really aware of them being totally wrong. Or perhaps just a bit ahead of their time.

While I think this is a neater solution, there is another product that does exactly the same thing, allow you to subscribe to channels and received pushed content via incentive compatible (you get faster speeds if you upload more) swarms.

It's called kast [sf.net].

Bittorrent kind of sucks

Realistically speaking, the biggest problem with Bittorrent is seeding. I think this is how bittorrent works:

* a file is seeded, and a .torrent file generated
* that .torrent file is uploaded to a tracker
* clients who want to download the file download the .torrent from the tracker
* the user opens the .torrent file, which causes the the bittorrent client asks the tracker for the machines/locations of the seeds and people downloading the file(s) pointed to by the .torrent
* the client downloads various chunks of the files from both the seeds and the other downloaders

The more people download a file, the better bittorrent is able to spread the bandwidth.

The downside is that if a file isn't seeded, it's no longer available. If a .torrent goes missing, the file is inaccessible. If the tracker goes away, the file is inaccessible.

Bittorrent's main problem right now, which is a client problem, is its upstream usage can easily swamp a home connection. That's just dumb client design.

Upload limiting works, but limits your download speed. The client develoeprs have to recognize that yes, sharing is nice and leeching is bad, but disrupting the users' connection is a Very Bad Thing.

Re:Bittorrent kind of sucks

Personally, I think that having to use a file to launch an application to join the torrent network is a bit convoluted.

I think that ideally, the most a user should see is bt://sitename.domain/file.zip, or something similar. The OS/browser should be able to handle that sort of protocol, and send it to the right application or use an integrated bit torrent client to get the file.

Correct me if I'm really wrong on this, or if it already exists. This would also be a welcome addition to Mozilla, I think.

Re:Bittorrent kind of sucks

Upload limiting works, but limits your download speed. The client develoeprs have to recognize that yes, sharing is nice and leeching is bad, but disrupting the users' connection is a Very Bad Thing.

A decent client, of which there are many, will let you throttle your outbound.

Note that the way torrent works, if noone uploads, noone downloads. And the faster everyone pushes, the faster everyone gets. Its not so much an artificial thing as it is an economy of bandwidth.

And again, if you don't like sending full tilt, find a better client.

Why RSS?

It's a shame they're using RSS, as it's a good idea with a bad implementation. There are currently 9 different versions of RSS, and all of them incompatible with one another [diveintomark.org]. It ought to be replaced with a better technology like Atom [intertwingly.net]. However, this does look like an interesting project, nonetheless.

What Bit Torrent needs more is:

Encryption. It should generate a unique key pair for each socket it opens.

Favorite Quote

My favorite quote in the article:

Subscription systems for large media files aren't new. Microsoft unsuccessfully attempted to build one, dubbed Active Channels, into Windows in 1997. But BitTorrent and RSS have garnered attention from some of the same media giants who abandoned Active Channels and other "push media" technologies in past years.

Gives me hope that corporations actually think before they make IT decisions and actually consider an open standard/protocol important.

Torrent Tracker Blog Plugin

Bingo! I built a tracker plugin for the php smarty base bblog system. I think the possibilities for this can be pretty damn cool. My .org is currently working on a media sharing platform for raw captured video for digital journalists. why restrict yourself to copying the crap from h(b)ollywood when you can hack it yourself on an imac? for info about the plugin (based on DEhacked)) and the super flexible bblog: http://bblog.com/viewtopic.php?t=392 you can use the bulit in rss of the blog to sync to your tracker...

IDEA: Bittorent on top of Webbrowser

How about a web browser with bittorent on top of it? this way, slashdot effect is a thing of the past