Become a fan of Slashdot on Facebook

 



Forgot your password?
typodupeerror
×
The Internet

Next Generation of Gnutella 70

ResearchBuzz writes "Wired is reporting the announcement of gPulp (general Purpose Location Protocol), an open source technology for search engines. From the article: "It is based on the Gnutella structure, an open source application originally created by Nullsoft....Using the basic protocols originally developed for Gnutella, gPulp will search for information across a network in real time.""
This discussion has been archived. No new comments can be posted.

Next Generation of Gnutella

Comments Filter:
  • by Anonymous Coward
    now i can get porn easy

  • The only way to solve the bandwidth issue is by aggressive caching of search results.

    I had an idea the other day, after reading the "Gnutella is dead" article -- why don't we build a multi-rooted tree system like the DNS, with information classified by content type and topic?
    Each category and sub-category could have autoritative servers that keep lists of filename/type/description -> URI mappings and pointers to authorities on more specific topics. All server on the way down (right down to the one running on your PC) can cache results until the TTL expires.

    Lists of "root" servers must, of course, be reasonably large and (geopolitically) diverse, and it must be relatively simple for anyone to express their intention to become an authoritative server for a given tpoic, otherwise the system becomes susceptible to censorship and regulation (as the DNS is at the moment).

    Anyone have any thoughts on this? It's only a rought idea and there are techical details that I haven't taken the time to think through (e.g. what would be the mechanism for registering your content with the authoritative servers?, how would the system defend against "polution"?, etc.)


  • "über-app" is gangsta?

    What side of the tracks do you live on buddy?

    Maybe if your name is Billy "the DoSman" Sanchez and you sit around drining 40's of diet 7-up then "über-app" is gangsta.

  • Compared to a centralized server for searching and a peer to peer sharing(napster style), Gnutella is definetly inferior. The extra network traffic generated during searches as the net is traversed is a waste and makes searches slow.

    However, if centralized systems such as Napster or even OpenNap servers come under attack (and we can see that they are) then FreeNet or Gnutella style services are the only alternatives.

    FreeNet is supposedly faster and more efficient than Gnutella. I personally find it ironic how campuses are banning Napster and students are resorting to Gnutella instead (which generates more useless traffic...)

  • I've been trying to find people interested in this, but nobody on the gnutella message boards is seriously interested.
    I've been toying with ways to use IM with public key authentication/encryption/etc, and such. I also know of a group working on rebuilding the IRCd protocol based on gnutella (for the servers only). Sorry, they don't have a web page.
  • Caching of results (and even files) is certainly a reasonable idea in such Peer to Peer systems.

    It does however start to create problems of its own. One feeling is that gnutella needs to be more anonymous. This would most likely be done by propagating a public key along with each search query and then accepting the results in an encrypted form - hence no intermediatary could cache results.

    The other difficulty with complex caching mechanisms is the potential for them to be badly implemented. Unfortunately these peer to peer systems rely on clients being properly written and properly configured. Gnutella still finds about 1/3rd of it's network traffic is consumed by a bug in the Pong response that caused any pong message to propagate to the entire network. If we start on with complex demand vs availability style caching then problems will no doubt emerge.

    The other problem with things like this is the possibility for abuse. Consider just what our DNS system would look like if anyone could add information to it - you'd end up with more entries like (sisgo.net - nslookup it - it's weird).

    Gnutella is already struggling to fight the script kiddies :(

    The only solution I can envisage would be some kind of star shaped topology. Take the basic gnutella network style and use it with people on t1+ connections to build a network backbone.

    Then onto them you can start hanging the cable modem and DSL users, and then each CM user can hold a couple of modem users. That MIGHT make it work.

    Then it would be the responsibility of the faster users whether or not to forward search queries... that way a search for something common like 'metallica' would never go further than the local group, but when u search for something bizarre it would go all over the network.

    The other way this could be extended is by having the slower users upload the lists of files they are sharing to the faster users where they could be cached.

    So we have a set of fast servers which are capable of answering on behalf of anyone else.... hang on that's napster.

    D'oh
  • I don't know how well or fast gPulp will catch on, but this type of protocol is exactly what we need.

    As mentioned in the past Slashdot article about Gnutella, these types of searches could be integrated in the hardware of routers, etc. I don't think that would happen though until a protocol was created & tested.

    So, hopefully this protocol or a similar one will be created, tested, and optimized, and then integrated into the hardware/software that runs the internet!

    I guess then Al Gore will get sued by the RIAA for creating the Internet as a music-stealing weapon of destruction.

    Rader

  • I don't believe peer to peer file sharing must inherently be less efficient than centralised file sharing. I think it's a matter of finding the right protocol. For one thing I think gnutella would benefit from caching and/or indexing. For example, nodes could be organised in supernodes, where each node of a supernode has an index to the files of all the nodes of the supernode.
  • Don't know why it was named after it, but there are a lot of people here in Denmark who like that stuff. And at least its not the most uninteresting name around. A little creativity and weirdness is a good thing...
  • If there is no demand for a piece of information, it will be removed from Freenet.

    I worry about this in Freenet too, from the other direction. It would be pretty easy to do horrible things to the performance of the network by inserting terabytes of spam (which is easily generated: you don't even have to store it all on your disk while you're doing it).

  • wait... dont stop. yer thinking in the right direction. napster's downfall was it's "purpose". you can share anything on gnutella. i share essays i've written and artwork and stuff. thats what we're really trying to build... a massive searchable distributed something or other file transfer network ... ya know?

    what we're really trying to create is the next generation of information sharing. face it...the http/ftp combination is out of date. this apprococh is more like having every single file on any anonymous ftp site and public web site distributed and searchable in such a way that every person who is intrested enough to download the info becomes a new mirror for it.

    get it?????

    james
  • by Vladinator ( 29743 ) on Monday September 18, 2000 @05:10AM (#771966) Homepage Journal
    ... search engines are also able to use this? gPulp/Gnutella clients alone will suck bandwith, but what happens when Yahoo/Lycos/AskJeeves, etc. see this as cutting into thier market share, and decided to do this searching for thier users? One plus to this might be that they could cache such hits, and could allow searching that didn't suck bandwidth, but with the imperminent nature of such connections, I bet they'd do a scan EVERY time someone looked for "Eminem*.mp3" at ftpsearch.lycos.com or some such.


    Fawking Trolls! [slashdot.org]
  • Are people really prepared to sit their for that long (probably a *lot* longer) in front of their browser for the results? Most research indicates that people who have to wait more than 5 seconds for a webpage to download get bored and go elsewhere.

    Napster searches routinely take 8-15 seconds for me (campus ethernet) and everybody else on my hall and nobody cares. When you're pretty assured you'll find what you want by putting in the right searches (as opposed to sorting links) people are willing to wait. The "web user getting bored" premise doesn't work here, mainly, because this isn't the web. ;)

  • Gnutella is starting to feel the same scrutiny that napster is. The corporations that they mentioned that they would like to form alliances with, will probably take a pass on making any partnerships. At best, IBM, MS, Sun, etc. will probably try to buy the rights to gPulp, and then release it under their terms, integrate it with SMB, etc. and make it stupid.
  • In ten years Gnutella, Freenet and similar subversive software will most likely lead into a situation where all the equipment required for a net connection will be licensed, traceable tamper proof black boxes. Protocols will enable transparent interception by the authorities and provide unique identification of the source, destination and even general content of each packet. Penalties for disrupting or circumventing the system will be severe and the legislation international.

    Governments and media just will not give up their accumulated power.

  • Not really. A search technology like this would practically eliminate the need for crawling robots since all the crawling happens intrinsically as part of the protocol.

    The impression I get from the article is that this could be used as a generic search technology.. one network that searches any type of resource i.e. web pages, file sharing, phone directories etc. It would just be a matter of writing a gPulp node server for whatever type of resource you wanted to add to the network.

    We will also need a standard result format so that the search client can figure out what to do with them automagically. We could use URLs, but types would have to be added for every kind of resource out there.

  • Only the French?I don't think so. [insidevc.com]

    And here we adhore that stuff (IANAF).
    Please respect our religion.

  • IMs would be like gnutella if they were distributed, unlike IRC or jabber which are not...

    *The server is the client.* Get it?

    Learn to read.
  • I do believe a well-built, user friendly P2P app will take the internet by storm.

    What makes you think so? I am not arguing that P2P is a good thing, but what makes you think that P2P+UI will be enough? Don't you think that there is something else implemented in Gnutella and/or Napster what makes them groundbreaking app? Like for example search features. This is not implemented properly yet. P2P -- well it isn't technology developed last year. How to search effectively -- that's the problem.


  • Sorry to be continuing this, but...
    wouldn't the operators of 'supernodes' be subject to legal action, thus compromising the whole reasoning for using a peer-to-peer system in the first place.

    Isn't this somewhat what FreeNet does? (caching, indexing, etc...) I've certainly heard that FreeNet improves on GnuTella quite a bit...

  • Well depending on how it was implemented, they could be, but unless they have a static IP or DNS entry it would be as difficult as any other gnutella user to prosecute.

    Taking off of the supernode idea I have one to offer..

    How about reversing the gnutella protocol. have two types of clients a standard client and an indexing client. the standard clients can only forge connections to indexing clients (except for file d/l). when a client connects to the index, it could upload an index of its shared directory(s) and all serches could be agains the index clients instead of the induviduals. Basically a client would tell the indexs that it is attached to what to search for and they would search their index and the indexs of other index servers they know. this would be a quasi centralized system, but stil lbe dynamic. Also it would limit the amount of ping/serach traffic that gnutella currently employs.

    Just a though but I think it is a good one

  • I'd like to point out that everybody in Europe eats the stuff, and I thought Americans also had it - c'mon, the stuff's *definitely* ten times less gross than peanutbutter and jelly (yes, I tried it once).

    Try this on for size : Peanut Butter and Jello (Jello - American for what the rest of the world calls jelly, the stuff that comes in crystals that you mix with hot water then stick in the fridge to set...)

    Don't laught; it works! (and it's also 10x better than PB+"Jelly")

  • by WolfWithoutAClause ( 162946 ) on Monday September 18, 2000 @08:51AM (#771978) Homepage
    The problem with the GNUTELLA protocol is that it it is quite inefficient, and it collapses completely at heavy load.

    The GNUTELLA protocol sends one message per search through the entire network upto the horizon.

    Eventually when enough people are in the network the individual links collapse under the load and the network falls apart.

    Nothing can completely solve this problem, but graceful degradation can certainly be designed for, and the way that the GNUTELLA protocol uses bandwidth can be very much improved, allowing for many more users, but at reduced horizon sizes.

    The current protocol wastes bandwidth in atleast one BIG way: it sends many short messages rather than one big one containing the requests.

    The reason that that is a waste is that each short message has a fixed sized overhead, at the TCP level. This means the useful percentage of bandwidth is significantly smaller than it might be if large messages were sent.

    Therefore it pays to hold off each search request for maybe 1 second before passing all the search requests on to the neighbours- the searching will be maybe 10 seconds slower due to the artificial delay (compared to 2 minutes; also offsetting this is the reduction in bandwidth), but possibly 50-100% as many users can be catered for.

    Secondly the behaviour at collapse can be much improved. To implement the above behaviour, each client should keep a list where it keeps requests/replies before forwarding them off to neighbours.

    If a request is held onto for too long without having a chance to pass it on then it should get thrown away (giving preference to low hop messages). That means that the horizon self tunes- avoiding collapse and giving better search performance at small network sizes.

    Otherwise death of the GNUTELLA net is predicted...
  • "Those French have a different word for EVERYTHING." - to quote Steve Martin
  • test
  • test2
  • test3
  • test4
  • test5
  • by Frijoles ( 16015 ) on Monday September 18, 2000 @04:33AM (#771985)
    From the article, you can find gPulp at http://gnutellang.wego.com/ [wego.com].
  • by zpengo ( 99887 ) on Monday September 18, 2000 @04:30AM (#771986) Homepage
    Pretty soon every C: drive will have to have a robots.txt file.

  • Peer to peer applications like this are just the beginning. As 'intelligent agent' software (little progs that sit on your 'puter and do stuff without much input or oversight on your part), and search/directory technologies such as this evolve, there truly will be a 'web' connecting every peer on the Internet, or on any given network for that matter.

    Peer to peer connectivity over distributed networks may well be the next killer app. since email.

  • I thought Gnutella's architecture was just found to be in danger of falling apart? [slashdot.org] All of the search requests having to traverse 56K conections, etc. Has this been corrected in the search engine.

  • remember the day when the distributed.net server was down?

    distributed not working
  • Can somebody tell me why the FSF named a piece of their software after a truly grim piece of food that only the French enjoy? Is this to be followed by GnuBovril, GnuMarmite and GnuGrits?

    ---
    Jon E. Erikson

  • Ooh! No more long nights spent wondering how to rearrange drive mappings to accomodate a new application without confusing users :)
  • Can anybody tell me why there is a load of animosity against the French? dons asbestos suit. Having laws designed to prevent their country being swamped by American 'culture' sounds like a good idea to me.
  • If there is no demand for a piece of information, it will be removed from Freenet.

    I'm not sure if this is good or bad - seems to defeat spam but if you've ever had to fill in 7 years worth of tax returns you'd be extremely interested to know when demand had reached 0.

  • Can Gnutella be reengineered to handle versioning more intelligently? Perhaps requests can be versioned.

    If we catch this soon enough, a (relatively) small population of Gnutella users may all be willing to upgrade.
  • Pretty soon every C: drive will have to have a robots.txt file

    Too bad they can just ignore them, *cough* *cough*

  • I guess the main intentive of this is to get rid of "network for copyright violation" stigma. So this is no more an app for sharing MP3s and warez, it's not Distributed Search Engine Platform. So you can now win those lawsuits to come, because Search Engine is good, while MP3 Sharing is bad (though the same thing is behind this, but most non-technical people never look behind the names).
  • Whatever happened to the idea of send a small program to a site to sift through its data and make intelligent choices about what to send back to the client?

    As opposed to the Gnutella, Napster, even HTTP protocols in which every request results in a huge number of hits and lots of extraneous file transfers.
  • I'd like to point out that everybody in Europe eats the stuff, and I thought Americans also had it - c'mon, the stuff's *definitely* ten times less gross than peanutbutter and jelly (yes, I tried it once).

    Oh, and to be on topic for a sec: maybe the searchengine should try to validate the sites that may be searched - in a starshaped network with the best connections on top...but I guess it might be expensive to set up local servers. Maybe it could be done Tucows style with local submissions or something :P
  • Go ahead and start your own root DNS, get other people to link their DNSs to yours and voila

    Can you say "META KEYWORDS"? Every search will lead to porn sites because they'll list "R/C Planes" and "Turkish Fashion" on their "interest sheets".

    This is a pretty good idea here

  • Having laws designed to prevent their country being swamped by American 'culture' sounds like a good idea to me.
    I wish we had some laws here in America to prevent us from being swamped by American culture.

    :^)

    ---------------------------
    "The people. Could you patent the sun?"
  • No one operates supernodes, supernodes are just a logical set of nodes that handle themselves by themselves.

    As for freenet, I don't know that much about it.

  • GnuVegimite. Yuck. Just as bad as Spam.
  • by Luminous ( 192747 ) on Monday September 18, 2000 @04:42AM (#772003) Journal
    When I read this story on Wired this morning, I was first very excited. I was thinking this had to be the groundbreaking p2p app. But when I saw it was based on the Gnutella structure, I recalled a previous discussion on Slashdot where it was said Gnutella had maxed.

    I am assuming when they say based on Gnutella, they have 'fixed' some of the problems. Or, maybe it really isn't searching the entire network, but key segments of the network. And maybe it just ignores 56k connections altogether.

    In your honest opinions, what is the viability of gPulp? Is this a bandwagon that deserves support or should we (okay, you real programmers) continue to develop a more robust and 'intuitive' P2P system. I do believe a well-built, user friendly P2P app will take the internet by storm. We've only scratched the surface. What I am afraid of is instead of looking down the road and seeing what the requirements and capacity of the P2P app will be, the development community will continue to add and tweak the current flawed or underpowered systems.

  • I'll be interested to see how they deal with the problem of congestion. I know that I hit search engines around fifty times a day. If each one of my (and everyone elses) queries were distributed a la Gnutella, then surely a significant proportion of total bandwidth would be used by those queries alone. Presently, each one of my queries creates a tiny amount of traffic. This is an efficient system. I realise the cost involved in hosting something like google, but google seems quite happy to bear it... I can only see gPulp being a success if it offered a quality of result not available with current systems. A system like gPulp should be fine for specialised searching (i.e. for phone numbers or mp3s) but I doubt it would be able to outperform the brute force of google in a more general case.
  • GPulp will also add a powerful location capability to networks.

    It's spelled "gPulp". gPulp isn't supposed to be capitalized, even at the beginning of a sentence. Didn't the editor read the jargon file?
  • From the article:

    Open source program developer Chris Childress thinks that gPulp could be the "über-app" that will completely change the Internet.

    Looks like someone is picking up their urban gangsta rapper lingo from CmndrTaco.

    ROTFLMAO.

    ---
  • by Mr_Silver ( 213637 ) on Monday September 18, 2000 @04:45AM (#772007)
    Lets hope that it doesn't suffer from the same problems as Gnutella.

    The technology of the Gnutella system is limited by the version that the majority use. In other words, if version 2.0 is out and has some cool new features, it'll be useless if the majority of people are using 1.0 because they won't recognise the new stuff.

    An example, say my version of gnutella client can do regular expressions. Throw a regular expression at another (non RE supporting) client and it'll either think its a normal search string (in which case you'll probably get nothing back) or it'll throw it away (so you'll also get nothing back). You can't win.

    You can't make people update, if you made v2 not badwards compatible them you'd fragment the network. Napster may be peer to peer but if they release a new client with new features then those people who download it get those features immediately (unless they require someone else with a newer client of course)

    My other concern is the spead of searches. Again the network is limited by the majority. With speed its going to be all those people on modems. Have they done any testing on a very large scale? Does anyone know how much faster the network would be without all the people on modems?

    In the end, either this will work and be groovy or it'll flop or it'll work for a while, overload and die.

    One final thought, you have to wait at least 20 seconds to get a decent number of results back. Are people really prepared to sit their for that long (probably a *lot* longer) in front of their browser for the results? Most research indicates that people who have to wait more than 5 seconds for a webpage to download get bored and go elsewhere.

    --

  • by Anonymous Coward
    We now hear that Freenet has matured to 0.3, while Gnutella is still chugging along. Do we really need so many "free" networks like this? Doesn't the use of two or more networks make information more difficult to find?

    Personally i believe that Freenet and Gnutella should join forces. If we can combine the two networks, we can double the size of it! Just think, Gnutella and Freenet together. There could be Terrabytes of information available to everyone who needed it.

    Even better, maybe someone should start working on an archive system for the two networks, providing a backup in the event that information begins to dissapear. Such a resource would be highly valuable to the users of Gnutella/Freenet. I'm sure it would be a huge undertaking, but i'm sure the costs could be covered by charging a small fee to retrieve information from the archive?

    What do other users think?
  • by warmcat ( 3545 ) on Monday September 18, 2000 @05:42AM (#772010)
    There are some very interesting proposals [wego.com] for the next version listed on the gPulp site.

    When I first saw it was ''talk before code'' my heart sank, but some of the proposals are actually very good.

    Also, I noticed at the Gnotella page [nerdherd.net] that the author is pointing out that Gnutella has being going downhill - no wonder, I find it much harder to get anything useful from it than with Napster.
  • "gPulp" is an unconventional spelling, and a trademark of sorts. Journalistic writing standards do not have to follow every silly-assed spelling gimmick. "Gpulp" would be the way a newspaper (at least until recently) would spell it.
  • by grahamsz ( 150076 ) on Monday September 18, 2000 @04:51AM (#772012) Homepage Journal
    Certainly in the context of using it on a Lan to locate things this could be a very powerful tool. However how many sysadmins would like to have to secure every workstation from hackers instead of just every key server... fun stuff.

    However on the internet it's doomed to failure. I followed the work on GnutellaNG for a little while and it seemed at that point to be involved in attempting to reduce the bandwidth requirements of gnutella by slimming down the protocol, whilst simultanously increasing the functions and hence bandwidth requirements.

    Ultimately any P2P system is limited by the outbound bandwidth that each user has. At the moment with about 3000 host on Gnutella you are using about 1.5-2kbytes/s for each connection you have open (most ppl have 2 - 4) plus that doesn't include bandwidth left to upload or download.

    Curiously though this would be the most optimum way of doing things (not gnutella in particularl but p2p) if it weren't for the fact that we have so little bandwidth at the end user.

    Even cable modem users typically have only 128kbit upstream, which will only take gnutella to about the 10,000 user mark before it starts to fall over again. The same has to be true of any raw peer to peer system.

    No amount of optimisation will reduce the bandwidth requirements of any search having to be executed on any host.

    Freenet on the other hand is a lot smarter than that and does actually move information about in a streamlined manner. Unfortunately I fear that freenet would fall over and die right now if it were holding the terabytes of files that gnutella does - so it appears not to be the best solution either.

    We need more bandwidth at end users and less at big corporations, except that would count as empowering the people and be morally repulsive to most politicians.
  • A few things I'd find interesting from gPulp, Freenet or what have you.

    IP tunneling - recreate a whole new web, allocate domain names freely and to first bidder - corporates will never have their say

    Interest sheets - by filling out an interest sheet for your node, the software connects tends to connect you to nodes sharing your interests thereby enhacing search retrieval speed on your topics of interest.

    Automatic load balancing - if I'm making a heavy use of non freenet services, reduces resources bandwith allocated to freenet. If I'm doing something else (or not using my computer)gives full bandwith to freenet, helps indexing/caching other nodes etc.

    On a side note, idealy, I think freenet should concentrate on making a secure encrypted overlay net protocol (IP tunneling, means of logging/using freenet anonymously) while having gPulp and the such concentrate on the file sharing.

    Please note, I'm not pretending to have a full grasp of the issues - these are just a few ideas.

  • So basically they're saying with a lot of words that gPulp will be a distributed search mechanism. Wow. Groundbreaking. The problem will probably not be getting it to work, but getting people to care. With free services like Google, there really isn't much reason for a general purpose distributed search system. Although there are several benefits, you have to convince the content providers to run the server themselves as opposed to just putting up content and letting somebody else spider it for them.

    (note: I am now refusing my +1 bonus because although I haven't been moderated down that much, Slashdot has decided to decay my karma)
  • Well - to be perfectly honest, the gnutella protocol needs drastic enough improvements that backwards compability is a pipe dream. Gnutella needs a lot of stuff - right now, it's not fit to handle what we throw at it. The idea of distributed searches in real time is kick-ass, though :)
  • The first thing that crossed my mind when I read
    about this, is - in the context of the location
    protocol -,

    'How is this different than the proposed
    IETF Service Location Protocol ?'

    http://www.ietf.org/html.charters/svrloc-charter .html

    Even if it is different (I did not see any mention
    on the gPulp site of an intent to make the
    location protocol compliant with 'SLP'), I fail
    to see why the need to invent another protocol
    that I believe fulfills the same need of the
    existing one.

    Arieh

  • Currently there are no real distributed operating systems that comply with all the requirements of such system. Many have attempted to use Solaris as such a system, however no Unix based OS can be called trully distributed. GNUtella, gPulp, FreeNet, HotLine etc. are only file searching and delivery systems but it seems that the logical step from their approaches should be the creation of a real distributed operating system, where all computers are resources of this OS and all users loged into one huge computer. If future versions of Linux could achieve this goal, Linux would become the final frontier of OSs
  • hmm... and I always thought that peanutbutter and jelly was worse than gnutella.. :P
  • They're running out of stupid puns starting in 'GNU'.
  • I imagine that solving the scalability problems would be the most looked-for feature in proposals for gPulp. After all, that's what it's all about - the project used to be called gnutella NG.
  • Um... the gPulp project is in an extremely early stage. Nothing about it is fixed, they are still asking for proposlas about the protocal.
  • ...unless these guys come up with an entirely new and better protocol.

    Ever notice how Gnutella just loves to suck all your bandwidth up, as twenty billion people search for "britney spears cumshot"? Well just imagine the joy of having the entire Internet glutted with that kind of crap.

    From their site, it looks as though they're still asking for proposals. So let's hope someone has better ideas than "Gnutella to the Stars!!!" if they hope anyone will run it on their servers.

    -=20

  • Does anybody see an IM application using Gnutella like protocols? Register yourself with your friends and maybe one of thousands of semi-static servers and totally negate the need for AOL/Netscape/Mirabilis and... who's left... Microsoft and Yahoo/Geocities?

  • but I haven't heard anybody address the fundamental problem with it: its vulnerability to attack. The best example is advertisers responding to every query with html garbage. But it seems to me that anyone who dislikes the service will do the same.

If you think the system is working, ask someone who's waiting for a prompt.

Working...