Next Generation of Gnutella 70
ResearchBuzz writes "Wired is
reporting the
announcement of gPulp (general Purpose Location Protocol), an
open source technology for search engines. From the
article: "It is based on the Gnutella structure, an open source application originally created by
Nullsoft....Using the basic protocols originally developed for Gnutella, gPulp will search for
information across a network in real time.""
oh goodie (Score:1)
Re:A bit exaggerated (Score:2)
The only way to solve the bandwidth issue is by aggressive caching of search results.
I had an idea the other day, after reading the "Gnutella is dead" article -- why don't we build a multi-rooted tree system like the DNS, with information classified by content type and topic?
Each category and sub-category could have autoritative servers that keep lists of filename/type/description -> URI mappings and pointers to authorities on more specific topics. All server on the way down (right down to the one running on your PC) can cache results until the TTL expires.
Lists of "root" servers must, of course, be reasonably large and (geopolitically) diverse, and it must be relatively simple for anyone to express their intention to become an authoritative server for a given tpoic, otherwise the system becomes susceptible to censorship and regulation (as the DNS is at the moment).
Anyone have any thoughts on this? It's only a rought idea and there are techical details that I haven't taken the time to think through (e.g. what would be the mechanism for registering your content with the authoritative servers?, how would the system defend against "polution"?, etc.)
Re:Another one? (Score:1)
What side of the tracks do you live on buddy?
Maybe if your name is Billy "the DoSman" Sanchez and you sit around drining 40's of diet 7-up then "über-app" is gangsta.
Gnutella == inefficient (Score:1)
However, if centralized systems such as Napster or even OpenNap servers come under attack (and we can see that they are) then FreeNet or Gnutella style services are the only alternatives.
FreeNet is supposedly faster and more efficient than Gnutella. I personally find it ironic how campuses are banning Napster and students are resorting to Gnutella instead (which generates more useless traffic...)
Re:Instant Messaging (Score:1)
I've been toying with ways to use IM with public key authentication/encryption/etc, and such. I also know of a group working on rebuilding the IRCd protocol based on gnutella (for the servers only). Sorry, they don't have a web page.
Re:A bit exaggerated (Score:2)
It does however start to create problems of its own. One feeling is that gnutella needs to be more anonymous. This would most likely be done by propagating a public key along with each search query and then accepting the results in an encrypted form - hence no intermediatary could cache results.
The other difficulty with complex caching mechanisms is the potential for them to be badly implemented. Unfortunately these peer to peer systems rely on clients being properly written and properly configured. Gnutella still finds about 1/3rd of it's network traffic is consumed by a bug in the Pong response that caused any pong message to propagate to the entire network. If we start on with complex demand vs availability style caching then problems will no doubt emerge.
The other problem with things like this is the possibility for abuse. Consider just what our DNS system would look like if anyone could add information to it - you'd end up with more entries like (sisgo.net - nslookup it - it's weird).
Gnutella is already struggling to fight the script kiddies
The only solution I can envisage would be some kind of star shaped topology. Take the basic gnutella network style and use it with people on t1+ connections to build a network backbone.
Then onto them you can start hanging the cable modem and DSL users, and then each CM user can hold a couple of modem users. That MIGHT make it work.
Then it would be the responsibility of the faster users whether or not to forward search queries... that way a search for something common like 'metallica' would never go further than the local group, but when u search for something bizarre it would go all over the network.
The other way this could be extended is by having the slower users upload the lists of files they are sharing to the faster users where they could be cached.
So we have a set of fast servers which are capable of answering on behalf of anyone else.... hang on that's napster.
D'oh
Re:Questions (Score:1)
As mentioned in the past Slashdot article about Gnutella, these types of searches could be integrated in the hardware of routers, etc. I don't think that would happen though until a protocol was created & tested.
So, hopefully this protocol or a similar one will be created, tested, and optimized, and then integrated into the hardware/software that runs the internet!
I guess then Al Gore will get sued by the RIAA for creating the Internet as a music-stealing weapon of destruction.
Rader
Re:Gnutella == inefficient (Score:1)
Re:Can anybody answer a question? (Score:1)
Re:Built in censorship (Score:1)
If there is no demand for a piece of information, it will be removed from Freenet.
I worry about this in Freenet too, from the other direction. It would be pretty easy to do horrible things to the performance of the network by inserting terabytes of spam (which is easily generated: you don't even have to store it all on your disk while you're doing it).
Re:A bit exaggerated (Score:1)
what we're really trying to create is the next generation of information sharing. face it...the http/ftp combination is out of date. this apprococh is more like having every single file on any anonymous ftp site and public web site distributed and searchable in such a way that every person who is intrested enough to download the info becomes a new mirror for it.
get it?????
james
How long before... (Score:3)
Fawking Trolls! [slashdot.org]
Re:However (Score:2)
Napster searches routinely take 8-15 seconds for me (campus ethernet) and everybody else on my hall and nobody cares. When you're pretty assured you'll find what you want by putting in the right searches (as opposed to sorting links) people are willing to wait. The "web user getting bored" premise doesn't work here, mainly, because this isn't the web.
Re:Instant Messaging "Yea, I do" -nt (Score:1)
I don't know about corporate sponsorship (Score:1)
In ten years... (Score:1)
Governments and media just will not give up their accumulated power.
Re:P2P Network Searching (Score:2)
The impression I get from the article is that this could be used as a generic search technology.. one network that searches any type of resource i.e. web pages, file sharing, phone directories etc. It would just be a matter of writing a gPulp node server for whatever type of resource you wanted to add to the network.
We will also need a standard result format so that the search client can figure out what to do with them automagically. We could use URLs, but types would have to be added for every kind of resource out there.
Re:Can anybody answer a question? (Score:2)
And here we adhore that stuff (IANAF).
Please respect our religion.
Re:Instant Messaging (Score:1)
*The server is the client.* Get it?
Learn to read.
Re:Questions (Score:1)
I do believe a well-built, user friendly P2P app will take the internet by storm.
What makes you think so? I am not arguing that P2P is a good thing, but what makes you think that P2P+UI will be enough? Don't you think that there is something else implemented in Gnutella and/or Napster what makes them groundbreaking app? Like for example search features. This is not implemented properly yet. P2P -- well it isn't technology developed last year. How to search effectively -- that's the problem.
Re:Gnutella == inefficient (Score:1)
wouldn't the operators of 'supernodes' be subject to legal action, thus compromising the whole reasoning for using a peer-to-peer system in the first place.
Isn't this somewhat what FreeNet does? (caching, indexing, etc...) I've certainly heard that FreeNet improves on GnuTella quite a bit...
Re:Gnutella == inefficient (Score:1)
Well depending on how it was implemented, they could be, but unless they have a static IP or DNS entry it would be as difficult as any other gnutella user to prosecute.
Taking off of the supernode idea I have one to offer..
How about reversing the gnutella protocol. have two types of clients a standard client and an indexing client. the standard clients can only forge connections to indexing clients (except for file d/l). when a client connects to the index, it could upload an index of its shared directory(s) and all serches could be agains the index clients instead of the induviduals. Basically a client would tell the indexs that it is attached to what to search for and they would search their index and the indexs of other index servers they know. this would be a quasi centralized system, but stil lbe dynamic. Also it would limit the amount of ping/serach traffic that gnutella currently employs.
Just a though but I think it is a good one
Re:Can anybody answer a question? (Score:1)
Try this on for size : Peanut Butter and Jello (Jello - American for what the rest of the world calls jelly, the stuff that comes in crystals that you mix with hot water then stick in the fridge to set...)
Don't laught; it works! (and it's also 10x better than PB+"Jelly")
The Gnutella protocol has big problems (Score:3)
The GNUTELLA protocol sends one message per search through the entire network upto the horizon.
Eventually when enough people are in the network the individual links collapse under the load and the network falls apart.
Nothing can completely solve this problem, but graceful degradation can certainly be designed for, and the way that the GNUTELLA protocol uses bandwidth can be very much improved, allowing for many more users, but at reduced horizon sizes.
The current protocol wastes bandwidth in atleast one BIG way: it sends many short messages rather than one big one containing the requests.
The reason that that is a waste is that each short message has a fixed sized overhead, at the TCP level. This means the useful percentage of bandwidth is significantly smaller than it might be if large messages were sent.
Therefore it pays to hold off each search request for maybe 1 second before passing all the search requests on to the neighbours- the searching will be maybe 10 seconds slower due to the artificial delay (compared to 2 minutes; also offsetting this is the reduction in bandwidth), but possibly 50-100% as many users can be catered for.
Secondly the behaviour at collapse can be much improved. To implement the above behaviour, each client should keep a list where it keeps requests/replies before forwarding them off to neighbours.
If a request is held onto for too long without having a chance to pass it on then it should get thrown away (giving preference to low hop messages). That means that the horizon self tunes- avoiding collapse and giving better search performance at small network sizes.
Otherwise death of the GNUTELLA net is predicted...
Re:Can anybody answer a question? (Score:1)
Re:GUIDS (Score:2)
Re:GUIDS (Score:2)
Re:GUIDS (Score:2)
Re:GUIDS (Score:2)
Re:GUIDS (Score:2)
gPulp location (Score:3)
P2P Network Searching (Score:3)
Peer to Peer (Score:1)
Peer to peer connectivity over distributed networks may well be the next killer app. since email.
Uh oh (Score:1)
i remember.. (Score:1)
distributed not working
Can anybody answer a question? (Score:2)
Can somebody tell me why the FSF named a piece of their software after a truly grim piece of food that only the French enjoy? Is this to be followed by GnuBovril, GnuMarmite and GnuGrits?
---
Jon E. Erikson
Local/intranet use? (Score:1)
Re:Can anybody answer a question? (Score:1)
Built in censorship (Score:1)
If there is no demand for a piece of information, it will be removed from Freenet.
I'm not sure if this is good or bad - seems to defeat spam but if you've ever had to fill in 7 years worth of tax returns you'd be extremely interested to know when demand had reached 0.
Re:However (Score:1)
If we catch this soon enough, a (relatively) small population of Gnutella users may all be willing to upgrade.
Re:P2P Network Searching (Score:2)
Too bad they can just ignore them, *cough* *cough*
Smart move (Score:1)
Use Agents to Reduce Bandwidth? (Score:2)
As opposed to the Gnutella, Napster, even HTTP protocols in which every request results in a huge number of hits and lots of extraneous file transfers.
Re:Can anybody answer a question? (Score:1)
Oh, and to be on topic for a sec: maybe the searchengine should try to validate the sites that may be searched - in a starshaped network with the best connections on top...but I guess it might be expensive to set up local servers. Maybe it could be done Tucows style with local submissions or something
Re:Wishes (Score:1)
Go ahead and start your own root DNS, get other people to link their DNSs to yours and voila
Can you say "META KEYWORDS"? Every search will lead to porn sites because they'll list "R/C Planes" and "Turkish Fashion" on their "interest sheets".
This is a pretty good idea here
Re:Can anybody answer a question? (Score:1)
---------------------------
"The people. Could you patent the sun?"
Re:Gnutella == inefficient (Score:1)
As for freenet, I don't know that much about it.
For the Aussies (Score:1)
Questions (Score:3)
I am assuming when they say based on Gnutella, they have 'fixed' some of the problems. Or, maybe it really isn't searching the entire network, but key segments of the network. And maybe it just ignores 56k connections altogether.
In your honest opinions, what is the viability of gPulp? Is this a bandwagon that deserves support or should we (okay, you real programmers) continue to develop a more robust and 'intuitive' P2P system. I do believe a well-built, user friendly P2P app will take the internet by storm. We've only scratched the surface. What I am afraid of is instead of looking down the road and seeing what the requirements and capacity of the P2P app will be, the development community will continue to add and tweak the current flawed or underpowered systems.
Talk about net congestion... (Score:2)
Spelling (Score:1)
It's spelled "gPulp". gPulp isn't supposed to be capitalized, even at the beginning of a sentence. Didn't the editor read the jargon file?
Another one? (Score:1)
Open source program developer Chris Childress thinks that gPulp could be the "über-app" that will completely change the Internet.
Looks like someone is picking up their urban gangsta rapper lingo from CmndrTaco.
ROTFLMAO.
---However (Score:3)
The technology of the Gnutella system is limited by the version that the majority use. In other words, if version 2.0 is out and has some cool new features, it'll be useless if the majority of people are using 1.0 because they won't recognise the new stuff.
An example, say my version of gnutella client can do regular expressions. Throw a regular expression at another (non RE supporting) client and it'll either think its a normal search string (in which case you'll probably get nothing back) or it'll throw it away (so you'll also get nothing back). You can't win.
You can't make people update, if you made v2 not badwards compatible them you'd fragment the network. Napster may be peer to peer but if they release a new client with new features then those people who download it get those features immediately (unless they require someone else with a newer client of course)
My other concern is the spead of searches. Again the network is limited by the majority. With speed its going to be all those people on modems. Have they done any testing on a very large scale? Does anyone know how much faster the network would be without all the people on modems?
In the end, either this will work and be groovy or it'll flop or it'll work for a while, overload and die.
One final thought, you have to wait at least 20 seconds to get a decent number of results back. Are people really prepared to sit their for that long (probably a *lot* longer) in front of their browser for the results? Most research indicates that people who have to wait more than 5 seconds for a webpage to download get bored and go elsewhere.
--
How many do we need? (Score:1)
Personally i believe that Freenet and Gnutella should join forces. If we can combine the two networks, we can double the size of it! Just think, Gnutella and Freenet together. There could be Terrabytes of information available to everyone who needed it.
Even better, maybe someone should start working on an archive system for the two networks, providing a backup in the event that information begins to dissapear. Such a resource would be highly valuable to the users of Gnutella/Freenet. I'm sure it would be a huge undertaking, but i'm sure the costs could be covered by charging a small fee to retrieve information from the archive?
What do other users think?
Re:Can anybody answer a question? (Score:2)
Sean
Re:gPulp location (Score:3)
When I first saw it was ''talk before code'' my heart sank, but some of the proposals are actually very good.
Also, I noticed at the Gnotella page [nerdherd.net] that the author is pointing out that Gnutella has being going downhill - no wonder, I find it much harder to get anything useful from it than with Napster.
Does not matter. (Score:1)
A bit exaggerated (Score:5)
However on the internet it's doomed to failure. I followed the work on GnutellaNG for a little while and it seemed at that point to be involved in attempting to reduce the bandwidth requirements of gnutella by slimming down the protocol, whilst simultanously increasing the functions and hence bandwidth requirements.
Ultimately any P2P system is limited by the outbound bandwidth that each user has. At the moment with about 3000 host on Gnutella you are using about 1.5-2kbytes/s for each connection you have open (most ppl have 2 - 4) plus that doesn't include bandwidth left to upload or download.
Curiously though this would be the most optimum way of doing things (not gnutella in particularl but p2p) if it weren't for the fact that we have so little bandwidth at the end user.
Even cable modem users typically have only 128kbit upstream, which will only take gnutella to about the 10,000 user mark before it starts to fall over again. The same has to be true of any raw peer to peer system.
No amount of optimisation will reduce the bandwidth requirements of any search having to be executed on any host.
Freenet on the other hand is a lot smarter than that and does actually move information about in a streamlined manner. Unfortunately I fear that freenet would fall over and die right now if it were holding the terabytes of files that gnutella does - so it appears not to be the best solution either.
We need more bandwidth at end users and less at big corporations, except that would count as empowering the people and be morally repulsive to most politicians.
Wishes (Score:2)
IP tunneling - recreate a whole new web, allocate domain names freely and to first bidder - corporates will never have their say
Interest sheets - by filling out an interest sheet for your node, the software connects tends to connect you to nodes sharing your interests thereby enhacing search retrieval speed on your topics of interest.
Automatic load balancing - if I'm making a heavy use of non freenet services, reduces resources bandwith allocated to freenet. If I'm doing something else (or not using my computer)gives full bandwith to freenet, helps indexing/caching other nodes etc.
On a side note, idealy, I think freenet should concentrate on making a secure encrypted overlay net protocol (IP tunneling, means of logging/using freenet anonymously) while having gPulp and the such concentrate on the file sharing.
Please note, I'm not pretending to have a full grasp of the issues - these are just a few ideas.
yawn? (Score:1)
(note: I am now refusing my +1 bonus because although I haven't been moderated down that much, Slashdot has decided to decay my karma)
Re:However (Score:1)
How is the location protocol different from slp ? (Score:1)
about this, is - in the context of the location
protocol -,
'How is this different than the proposed
IETF Service Location Protocol ?'
http://www.ietf.org/html.charters/svrloc-charte
Even if it is different (I did not see any mention
on the gPulp site of an intent to make the
location protocol compliant with 'SLP'), I fail
to see why the need to invent another protocol
that I believe fulfills the same need of the
existing one.
Arieh
Distributed Operating System (Score:2)
french bad taste (Score:1)
Simple answer: (Score:2)
Re:Uh oh (Score:1)
Re:Questions (Score:1)
This will never fly... (Score:2)
Ever notice how Gnutella just loves to suck all your bandwidth up, as twenty billion people search for "britney spears cumshot"? Well just imagine the joy of having the entire Internet glutted with that kind of crap.
From their site, it looks as though they're still asking for proposals. So let's hope someone has better ideas than "Gnutella to the Stars!!!" if they hope anyone will run it on their servers.
-=20
Instant Messaging (Score:2)
Does anybody see an IM application using Gnutella like protocols? Register yourself with your friends and maybe one of thousands of semi-static servers and totally negate the need for AOL/Netscape/Mirabilis and... who's left... Microsoft and Yahoo/Geocities?
I love the idea behind GNUTELLA (Score:2)