Follow Slashdot stories on Twitter

 


Forgot your password?
Close
typodupeerror
×
The Internet

Bridging Torrent and RSS 81

Posted by CmdrTaco from the working-out-the-kinks dept.
lerhaupt writes "PEP, the Prodigem Enclosure Puller, is a small php script which find all the enclosures in an RSS 2.0 feed URL, and utilizing Prodigem's new bittorrent API will have a torrent created and seeded for each. As an example of just what this exactly means, Prodigem is now using PEP to automatically torrent the top items found in the del.icio.us popular video feed. In general this now means distribution via bittorrent can be had with almost zero work or duplication of effort."
This discussion has been archived. No new comments can be posted.

Bridging Torrent and RSS More

Bridging Torrent and RSS

Comments Filter:

  • PEP source code (Score:5, Informative)

    by NoInfo ( 247461 ) * on Wednesday September 21, 2005 @09:19AM (#13613407) Homepage Journal
    PEP is less than 400 lines of PHP. Here's the source code for the curious:
    http://prodigem.com/code/pep/pep.txt [prodigem.com]

    (from the PEP home page)
    • Thank-you very much for the link to the source code. I am not a coder, the closest I get are simple bash scripts to automate stuff on my servers.

      2 things that I noticed while looking at that file;

      (1) how awesome GPL really is. Sprinkled throughout are snippets of code that was borowed from other sources (with links back to original docs).

      (b) Why do differenct programming languages use different codes to "comment-out" areas?

      It wasn't untill 6 months ago that I found out REM doesn't work anymore!!

      So we have
    • Can we get this on the slashdot front page? Article links get cached through bittorrent, and with the slashdot effect, we will be downloading the files lightning fast and freenet-style latency won't even be a problem with a custom tracker specialized for serving small files. Of course, it doesn't work when nobody reads the articles anyway.. :)

  • anybody... (Score:5, Funny)

    by Anonymous Coward on Wednesday September 21, 2005 @09:20AM (#13613414)
    have a torrent for this?

    seeds please.

  • found in the del.icio.us popular video feed

    I can't see any videos there, should be any? It's just a compilation of popular del.icio.us links, just like Oishii [opencontent.org] but built internally.

    --
    Superb hosting [dreamhost.com] 4800MB Storage, 120GB bandwidth, $7,95.
    Kunowalls!!! [kunowalls.host.sk] Random sexy wallpapers (NSFW!).

    • Re:Video? (Score:4, Informative)

      by Thuktun ( 221615 ) on Wednesday September 21, 2005 @10:17AM (#13613923) Journal
      Superb hosting 4800MB Storage, 120GB bandwidth, $7,95.
      Kunowalls!!! Random sexy wallpapers (NSFW!).

      Would it be asking too much for you to keep your sig to your sig?
      • The new rage these days is pasting your sig into the post, so that astroturfers can put as much stupid advertising in as they want. Plus, you can't just turn sigs off to get rid of them. It's a win-win! Oh, wait.

  • So, for us who can't code (Score:3, Funny)

    by RealisticCanadian ( 850967 ) on Wednesday September 21, 2005 @09:26AM (#13613482) Journal

    Am I correct in believing we will soon have an even greater wealth of torrents readily available to the users of the World?

    If so, three words for the *AA's: Yer Fucked. Cope.

    :^D

  • Hmm... (Score:5, Insightful)

    by FlyByPC ( 841016 ) on Wednesday September 21, 2005 @09:31AM (#13613529) Homepage
    Automatically-downloaded RSS, distributed over the Net;

    Automatically-initiated Bittorrent downloads;

    ...Is it just me, or with the addition of a few local scripts, does this sound like a virus-writer's favorite fantasy? They need to make sure there are some very good safeguards on this!

    Then again, with Genetic Algorithms, perhaps this is the next missing ingredient for cybersentient life...

    • Re:Hmm... (Score:5, Informative)

      by glesga_kiss ( 596639 ) on Wednesday September 21, 2005 @09:44AM (#13613650)
      This was done like a year ago and there have been no issues so far. Remember, most folk will be downloading non-execuatable media, and any binaries will not be automatically executed on download. Downloading pirate binaries has always been "at your own risk". Really, this is no more dangerous than the same number of users manually clicking the link. Neither way is less vunerable WRT viruses.

      Azureus and other BT clients already have RSS readers, using Regex's to match media in the RSS xml. This rules for TV series that are currently airing, it's great to return home to two or three new episodes each day. Many BT sites have RSS feeds, however the only one that got it right was btefnet, who had the inteligence not to post the same media twice. Most other sites have moron users who will post a 200 meg version of the file, followed by a 50 real media rip that no one wants but wastes our bandwidth because we get it anyway!! Throw in ratio sites, users dropping in 4 gig "season 1 dvd rips" and you can see why I stopped using it in the end. We just need a BT site that "gets it". Answers on a postcard please...

      • Re:Hmm... (Score:1, Informative)

        by Anonymous Coward
        They're out there, but much less high profile than btefnet.
      • To solve or party solve this problem is to couple it with a service such as orkut. Then we could certain users who we would automagically download from. So when a friend of mine on orkut who likes sitcoms finds/uploads a good show, it would get it from him. He would be careful not to upload dupes. Even better still would be an email or SMS that would allow me to choose which ones I would like to DL : )
      • An easier solution is to just write something on the client side. Something that maybe scrapes all of the feeds, uses intelligent parsers to group together all of the dupes (and ignore entire season releases, etc), and then uses some selection criteria to grab the best quality (video quality, number of seeders, etc) torrent out of a pool of dupes. Match that with simple filter syntax (no need to use RegExps... series titles are 'normalized' behind the scenes, and matched that way), and you have a quick an

      • The "RIGHT WAY" (tm), if you ask me, would be to have a bt site that offers a subscription to whatever "channel" you want to get distributions from. For example, if you want to make sure that you get in on the ground floor when the next Ubuntu stable is released, you should go to Ubuntu's site, download a tiny little file, and drop it into your hypothetical RSS/Bittorrent hybrid program (or even better yet, just cut and paste a URL or whatever into the program, which it then checks for updates). Then, whe
      • We just need a BT site that "gets it". Answers on a postcard please...

        Yo, let's start one! Slashdot style moderation of torrents, karma and whatnot. Send me an e-mail! ;)
    • "...Is it just me, or with the addition of a few local scripts, does this sound like a virus-writer's favorite fantasy? They need to make sure there are some very good safeguards on this!"

      RTFA, it wouldn't spread under any normal implementation.

  • Just another pollutant (Score:3, Insightful)

    by rahlquist ( 558509 ) on Wednesday September 21, 2005 @09:36AM (#13613576) Homepage
    Much like breathing outside, having scripted automatic downloads of 'anything' in a feed isnt a great idea IMNSHO. Sounds like walking out the front door and having the exhaust from a diesel blown in my fact just because I said I liked trucks.....
    • However, rigging it to automatically torrent any movies or sounds linked in a /. article would be a good idea, hmm?

    • Re:Just another pollutant (Score:4, Insightful)

      by Bogtha ( 906264 ) on Wednesday September 21, 2005 @10:23AM (#13613971)

      having scripted automatic downloads of 'anything' in a feed isnt a great idea IMNSHO.

      Did it ever occur to you that feeds themselves are scripted automatic downloads?

      • Did it ever occur to you that feeds themselves are scripted automatic downloads? Allow me to expand. The feeds are tiny in comparison to the average podcast or video stream. A RSS feed is chosen by a conscious decision to download a little bit of data at a regular interval. Joe Schmo sets something like this up and we could have 1000's of people downloading gay p0rn when a web site gets hacked, etc..

        • A RSS feed is chosen by a conscious decision to download a little bit of data at a regular interval. Joe Schmo sets something like this up and we could have 1000's of people downloading gay p0rn when a web site gets hacked, etc.

          Firstly, if somebody hacks a website, they can put gay porn on it anyway. This script doesn't change that.

          Secondly, if you are objecting to the fact that people can be tricked into downloading unwanted videos instead of simply unwanted pictures, then a) you can do that with

  • Terrible writeup (Score:5, Informative)

    by Bogtha ( 906264 ) on Wednesday September 21, 2005 @09:43AM (#13613632)

    Okay, to clarify a few things:

    Enclosures are basically the RSS way of providing a link to an external resource instead of having a normal entry. Think podcasting - basically an RSS file with links to MP3s instead of textual entries.

    What this tool seems to be intended to do is take an RSS feed, download all the external resources from it, then generate and seed torrents for each external resource.

    For those of you thinking that this is a way of distributing RSS feeds via BitTorrent, think again - the feeds are distributed normally, and this doesn't let existing feed readers do anything new with BitTorrent, they'll still be downloading both the feeds and the external resources though HTTP.

    So basically, this would take a podcast, download the MP3s, and generate/seed torrents for each of the MP3s. The torrents would then appear on this PHP page for people to download, but feed readers wouldn't know anything about it and carry on operating exactly the same as normal.

    • Just to clarify, Prodigem comes with a torrent feed for each user. So once all your enclosures have been converted to .torrents, you can provide people with your new feed. For example, http://www.prodigem.com/torrents/rss/pep_delicious .xml [prodigem.com]
    • Could a site have an enclosure that referred to a torrent feed? Would any readers support that?

      I think that's what will really make torrents take off, when in the browser and elsewhere torrents are supported internally so you click on a link and the video (or whatever) just starts downloading, sharing the torrent while the download is in-progress. I thought Firefox was working on that in fact...

    • For those of you thinking that this is a way of distributing RSS feeds via BitTorrent, think again - the feeds are distributed normally, and this doesn't let existing feed readers do anything new with BitTorrent, they'll still be downloading both the feeds and the external resources though HTTP.

      But when your RSS Feed can reference its own ".torrent", couldn't your news aggre-reader automatically hand that off to a BitTorrent engine underneath?

      The perfect world: LiveCDs with Lots of RAM and no permanent stor
      • You probably could distribute RSS via torrent, but i dont think you would gain much. You still need to fetch the .torrent from the rss webserver, so it is still getting lots of connections. Sure, the .torrent will be smaller than an xml file, but using the apache gzip should make them closer and wouldnt need a tracker or torrent client seeding.
        • You're using a lightweight protocol (RSS) to pass around small bits of data (.TORRENT)

          Using Enclosures isn't the way to do it -- rather, you should be including that content in-line.

          But "linking" to a shared resource is a lot more like what the "Enclosure" tag was meant to be. In effect, if you replace the "http:" with "torrent:" your Enclosures would be that much more efficient.

          What you're *NOT* doing is sending creating ".torrent" files of RSS content. What would be the point? All that XML is short-liv

  • EPIC 2014 (Score:4, Insightful)

    by USSJoin ( 896766 ) on Wednesday September 21, 2005 @09:48AM (#13613681) Homepage
    This, IMHO, totally rocks. Seriously: the computer now handles its own slashdot effect, intelligently. Truly, this allows for media-centric societies (like EPIC 2014) to be propogated; when we don't get that "Too many connections, MySQL fails miserably" message with popular content (even on normally-unpopular webpages), we are able to much more quickly diseminate information of interest to all of us.

    Kudos to the developers; I, for one, am impressed.

  • So it nearly happened... (Score:5, Funny)

    by samjam ( 256347 ) on Wednesday September 21, 2005 @10:12AM (#13613870) Homepage Journal
    Someones job has been replaced by a small PHP script.

    Shell script to follow :-)

    Sam

  • Oh, How The Litigators Are Gonna Love THIS (Score:5, Interesting)

    by ausoleil ( 322752 ) on Wednesday September 21, 2005 @10:19AM (#13613941) Homepage
    Automation of Bittorrent was inevitable. More or less, torrents become the functional equivilant of the automated podcast gathering programs.

    Things is, most podcasts are original content. Much of the 'torrents, and let's be honest, are not, and they are not exactly sanctioned (meaning they're pirated works.)

    Gee, do you think that the MPAA legal goons will be among the early adopters? Think that they will have the RIAA folks for company?

    Bet the farm on it.

    Come to think of it, one must wonder when or if the adult industry will resort to infringement lawsuits to protect their unique content...sure there has been a scant amount of it, but eventually, someone is going to pay big for those Jenna Jameson clips. (LOL.)

  • I would like to envision a future internet... (Score:2, Insightful)

    by Anonymous Coward
    ... where virtually _all_ content that is viewable on the web is distributed in the same manner as bittorrent content - where your web browser shares all content (including images, and any other embedded objects that may display, such as video or audio [with the exception of realtime live streams]) that is on the web page(s) that the browser is open to (usually uploading the whole time the reader is spending viewing the page). Browsers would contact the server to initially request the page, and then conta
    • Do you realize how long it takes for my BT client to actually contact and connect to enough peers to start downloading a file at greater than dialup speed? I can download the entire /. front page before that happens.

      And you want me to do that for every single image on a site? I'd go insane!
      • The slashdot effect that I believe the grandparent post is referring to does not affect slashdot itself, so much as it affects the loading speed of pages with large amounts of content, such as high resolution images, or videos, which are typical for pages that end up experiencing the slashdot effect when they are linked to from stories here.

        And in all fairness, if all web browsers observed the protocols as the grandparent described, it wouldn't take long to contact enough peers to start downloading at all

  • SwarmStream? (Score:5, Interesting)

    by Doc Ruby ( 173196 ) on Wednesday September 21, 2005 @11:00AM (#13614304) Homepage Journal
    Does anyone know how to make a swarm network like Torrent deliver chunks in a more sequential order? Even if locally they're unpredicted, maybe a "superchunk" order, where, say, each megabyte arrives in order, but the kilobytes within it fill in in an arbitrary order. That's what's necessary for Torrents to stream media like video on demand. If there are enough servers in the swarm (say millions), enough will likely be online at any time (say thousands) with the chunks that enough sources can respond (say hundreds) to ensure that every superchunk can be delivered "on cue" (pun intended), delivering a stream with minimum accumulation latency that can be compensated by a few-megabyte (milliseconds) buffer.

    • Re:SwarmStream? (Score:3, Informative)

      by shadowmatter ( 734276 )
      Try CoolStreaming. Although I haven't seen it in action, some of my friends have (mostly while visiting Europe or Asia), and they say the quality is near-perfect. And this isn't a stupid academic exercise -- it's a real implementation, with up to 10,000 simultaneous users recently. The academic paper, providing the general algorithm, can be found here [cs.sfu.ca]. Google for more on the implementation.

      - shadowmatter
      • It all sounds pretty exciting. I just hope that the CoolStreaming.org website [coostreaming.org] isn't representative of their SW's quality. Because it's badly broken in so many ways. Apparently "they've ended their experiment" [blogspot.com], but that's no reason to leave the lab bench cluttered with broken detritus. Which suggests that perhaps there's a more public-oriented org waiting in the wings to package this new protocol right - or maybe just these guys on their next reemergence.

    • Re:SwarmStream? (Score:4, Interesting)

      by SlayerofGods ( 682938 ) on Wednesday September 21, 2005 @11:31AM (#13614552)
      You can do this in bittorent, you'd just need to tell your client to grab the pieces in order (I don't know if there are any that do this... but it would be possible to program it to do so)
      Problem is however it will seriously degrade the ability of the network to provide the last pieces if everyone does this since that's when it's most likely someone will D/C.
      Using random pieces ensures that the network will have a good amount of all the pieces and not a lopsided amount of the pieces towards the start.

      Steaming over a swarm is problematic anyway because of varying connections speeds. You'd need to buffer so much ahead in case you hit someone with a slow DL speed or someone D/Cs in the middle of you downloading that chunk that it would basicly be just like downloading it anyway ;)

      • Re:SwarmStream? (Score:3, Interesting)

        by Doc Ruby ( 173196 )
        Does the BT protocol already have a "mode" or config that already tells server peers to deliver packets in order? Or is the reprogramming you're talking about something that would work only with all reprogrammed torrent agents at opposite ends of the connections?

        FWIW, the "superchunk" protocol architecture I described is designed to address exactly that latency problem, which is the entire shortcoming of existing torrent apps.
        • The reprogramming would only need to be done at your end.
          Basically right now your client program just requests the files in random order (well quasirandom because I believe it also uses the availability of a chunk to determine when it should download it) so they're delivered in random order.
          All you'd need to do is tell your client to ask for the pieces in order. Simple as that.
          But like I said before... even though it's possible, it could (will) hurt the network if it becomes popular; so there probably (hop
      • You would want it to prefer the in-order chunks, but if the availability of chuck X++ is two low (slow, availability issues) then you would fall back to the generic BT random order.
      • http://www.peercast.org/ [peercast.org]

        This does the whole live-stream multicast over P2P thing. Suprisingly, it sometimes works.
    • http://participatoryculture.org/bm/ [participatoryculture.org] is somehow able to do that with torrents.
      also not exactly what you asked.

      with azuerus, if it's multiple files, you simply set a priority on the first file, and those chunks will come first (in the view files in the torrent section.)

      It would be much more useable if that was included in the torrent seed file (possible?).
  • Excellent. We're on the verge of a universal distributed file system that's scalable and resilent.

    Now we just need a sensable self-organization scheme so we can accurately locate things.

    • I think you seriously overestimate the features of this little toy.

      There's quite a long step from a program that republishes files it finds on the internet and a global filesystem. In fact, they're almost nothing alike.
  • Look, there are at least 12 other BT + RSS clients. A dozen. The idea has been done. About a year ago I put together a list [asyserver.com] which was canonical at the time, but at this point, I know of at least three others that I hadn't gotten around to adding to the list.

    So, although I've got nothing against this one, it's not as if having one more client changes anything. The article makes it sound revolutionary. If nothing else, both Torrentocracy and Videora have been posted on Slashdot previously.

    Keith

    • I'm sorry. I misread the article. This isn't about the client end, its about the server end, and good, easy-to-use solutions for that are basically non-existent, so this is a step in the right direction.

      Keith

  • Yawn (Score:2)

    by volkris ( 694 )
    Yawn.

    I wish there was a way to mark a slashdot posting with "uninsightful." Why is it that almost every single bittorrent-related posting to Slashdot is completely lacking in any sort of real advancement?
  • As an analog of prodigem's automatic torrent creation, http://isohunt.com/ [isohunt.com] provides search feeds over RSS of most BitTorrent resources over the internet. So you can search for a TV show for example and be updated on the newest eps in the RSS feed.

    Cheers

Slashdot Top Deals

Enzymes are things invented by biologists that explain things which otherwise require harder thinking. -- Jerome Lettvin

Close