Peer-to-Peer Search Engine Wants You To Help Grub 64
FuzzyMan45 writes: "Check this out! Grub.org has finally finished writing their internet crawler. For those of you who don't know, grub is a distributed internet crawler that is indexing the internet and working towards an almost realtime index of all the pages combined with a search engine. Think about it, a search with no dead links and no out-of-date pages! Grub is the way to go." It sounds like a cool hybrid of client- and server-side information (a crawler works from your computer, with updated findings sent to a central repository) and all GPL'd. If grub outperforms Google, I'll be happy as a google-using clam -- but unless they have google's logic and caching, that is a very tall order. A better search engine is a pleasant dream though.
Re:Google (Score:1)
Its the new best thing.
Also goggle.com [goggle.com] seems promising but they don't have many pages *yet*.
This won't work (Score:1)
Besides, the real issue with search isn't crawling. It's indexing and ranking. They don't seem to have any real plan for how they're going to do that...
The name grub is already taken [uruk.org]. I suggest the name 'maggot' as an alternative. This name would be very appropriate, because for the most part it's going to be crawling the rotten meat of the web.
Kascade (Score:1)
Please visit the project website [kascade.org]and tell me what you think.
Re:They say they WILL provide a searchable index.. (Score:1)
Free work, bad indexes and huge loads? (Score:5)
1) Lots of people running grub means that they don't have to spend money on bandwidth to index sites, they get nice indexed data for free. Sure it's nice for them, but it seems like a kind of lame thing to do, I wonder how this will set with people.
2) What if I modify/run their GPL software to index my collection of web pages, and I return indexes that contain pages that aren't really there to taint their search engine with my 10,000 non-existant pr0n pages. Do they have a way around that problem?
3) Userland's editthispage.com and weblogs.com stopped allowing certain bots to index them because they were being abusive. Google was accidently blocked, but they were being nice. But still, the load of Google crawling all those pages is huge. Grub says they can crawl every page on the internet every day. What kind of load will this cause? The faq says your client will only do part of the internet, so you won't have everybody hitting your site daily, but I'm wondering what kind of flaws exist in that logic. Could this result in huge loads on webservers like the slashdot effect or a DDOS?
What's the license on the database? (Score:5)
--
What about manipulation? (Score:3)
The grub concepts sounds good, but I doubt it will stand reality unless you create a complex "web of trust" system. (Which in turn would be too complex for grub to become popular.)
Re:Free work, bad indexes and huge loads? (Score:2)
1) Some webmasters go to great lengths to ensure their pages are spidered often, whether it is nefarious attempts to get first page listing or just attempts to keep the engines up to date with content. These people would be happy to run a client that kept their page listing fresh.
2) Count on it happening.
3) Userlands content management system could do better to recognize robots and present them with a narrower view of the site. I set a robot to pull down one of my editthispage sites and it found every version of each page that Frontier can generate, at least 20 versions of each page. But you're right, what if grub starts spidering every morning at 8am, we're going to have some bad net congenstion.
I would imagine some of this has been addressed but these are important issues.
Chris Cothrun
Curator of Chaos
_I_ don't want to lose Google's _cache_ (Score:2)
Re:What about manipulation? (Score:2)
Hrm idea.. why couldn't you hook up a spider that searched the Net in this distributed way and added all files to FreeNet along w/ some sort of search system. Wasn't there something about an SQL engine for FreeNet posted the other day? That sounds interesting.
My Powerbook kicks ass (Score:2)
Re:What about manipulation? (Score:2)
Or worse, pretend to be pr0n just to lure people in. That would be much worse - nobody's more impatient than when they're cruisin' for porn :)
Caution: contents may be quarrelsome and meticulous!
Re:Problems with Google (Score:2)
There's a solution [google.com].
GRUB is a Bootloader. (Score:2)
It's ridiculous... GRUB is a bootloader,
not an url indexer.
Worse than that: I've built their client,
and it even clashes with binary names.
The client binary is called 'grub', which
clashes with my
See also:
http://www.gnu.org/software/grub/
Please hand over the grub.org domain
to GNU, and stop confusing me.
Bram Stolk
Re:Free work, bad indexes and huge loads? (Score:3)
Ultimately this comes down to the fairness and who owns the database itself. If its open and free to everyone, then this is a good cause, even if they have to generate revenue to support the site that serves the searches. In fact, if they distribute their servers properly, it won't even be necessary to have a large revenue stream as the load could be scattered more or less evenly over the entire volunteer pool.
Unfortunately, due to their somewhat fishy intentions with regards to revenue, it might end up killing the project before it ever takes off.
And it really seems like a good idea too.
Oh well.
-Restil
This is not Stallman (Score:1)
Re:You've just taken the bait... (Score:1)
Wow, it seems that you also have a problem with the guys that sell Resin or with all of the Linux distros. :)
Does Grub want to make money? Yes, there's no way Grub can afford the overhead of the service without passing the costs on to someone somewhere. However, that in and of itself does not make Grub an opportunist. In this current market, investors couldn't care if you claimed your project was powered by Jesus if it didn't have a solid business model under it.
The reasons Grub is opensource are varied. Kord comes from an ISP and BSD background. He knows from his own experience that a sys admin is not going to install networked software on a box when that network software can't be inspected. Having Grub be open source is insurance for the wise sysadmins. Kord being a FreeBSD freak from the start, I'm pretty sure closed source doesn't do it for him. Also, if you analyze the business model, there's no compelling reason to make it closed source. The money is in the data, not the ins and outs of how the clients and servers work.
Some Clarifications on Grub. (Score:4)
Wow, I'm surprised to see Grub on Slashdot this morning. The first client beta was _just_ released last night!
Anyways, I know the Grub guys and was there when Grub was just an idea being discussed over coffee. Although I can't speak with 100% authority, I feel that I can give some insight and perhaps some clarification to a few concerns/questions floating around. It appears that Kord and Iggy may have left a bit to be desired on the FAQ :)
From my understanding, the initial desired audience is the ISP admin. As an ISP, you'd be able to have your grub client index and crawl sites that you host. In turn, those sites will be available on whatever search engine Grub is supplying data to. Those running an ISP or hosting websites know how often clients request that you make sure they get crawled and listed in a search engine; this is a pretty nice value-add for your ISP service then. In this case, it's a win-win scenario. Grub gets up to date information on sites and the ISP gets to provide a much requested service to its customers.
Later, I believe the plan to encourage individuals on broadband connections is to provide rewards for a certain number of sites crawled and also prizes for top crawlers.
There are some concerns about the licensing of the database. It's my understanding that Grub is taking a commercial-pay/non-commercial-free approach. That means for instance, if you started an opensource search engine like aspseek.org you could use the Grub data for free. But if you're Google or Inktomi, you'll have to pay for access.
The data will not be free to everyone. There's just no way anybody can provide the overhead costs for that kind of service free to everyone. I think charging only for commercial use is the best option in this case. Also, keep in mind that the server will eventually be released as well. This means that individuals could run their own grub servers and stockpile their own data.
As far as the few statements regarding the stock options payment, I'm pretty sure all of the in house full-time developers get paid real money. However, Kord is really determined to make sure that those people kicking in 5-10hrs a week in their spare time get to share in some of the success when Grub hits it big. Once again, that's a win-win situation. The contributors get to work on a promising, useful OS project and if the world comes knocking for this better mouse trap, the contributors also get a bit of cash for their troubles.
I'd encourage those that have concerns or are curious about the project to go ahead and download the client now while it's in such early development. Take a look at the code. Email Kord and Iggy and tell them what you think. Even email them if you think Grub is a stupid idea, but tell them why. I don't think wanting to make a successful commercial P2P application is a bad idea in and of itself.
Interesting business plan (Score:3)
-russ
Mirror... (Score:1)
Re:No winblows client (Score:1)
We plan to port a release to Windows within a few months.
Hey! Bill says you can't do that! (Score:1)
Good luck to them, I think we need a few companies making extortionate profits from free software. I won't be queuing up to help them make a buck, but I will be expecting them to add to the software base.
Xix.
> Fourth, Grub will provide consulting services
> for companies wanting to set up their own Grub
> networks. Large corporate intranets could be
> quickly and efficiently indexed into a central
> database with the Grub client/server model.
> Consulting and coding for these proprietary
> installations is a common model in Open Source
> oriented businesses like Sendmail, MySQL and
> Apache.
Re:Why bother? (Score:1)
More info in my next post.
Kordless
A Grub's Response (Score:5)
First off, we didn't expect to get Slashdotted so fast. We really weren't ready, but we do appreciate the attention that it has brought. I'd like to address some of the issues that you guys have brought up because they ARE important to us. After all, you are (hopefully)running the client and we do care about what you think.
OK, about the money thing. A few of you are blasting us for trying to make a buck off this idea. Last time I looked, it takes money to pay for servers, bandwidth and programmers! We didn't start this thing with the intention of ripping people off - we did it because current search engine crawling technology is behind the times and we thought we could fix it. Don't fault us for having a revenue model and a desire to build a solid company that feeds us.
A LOT of you guys could benefit from having your web pages continuously indexed. It would help your customers AND it possible could increase the quality of service you provide. Besides, we aren't proposing to charge you for this service if you help out by running the client - that would defeat the purpose of the whole project.
Don't you already pay for bandwidth that gets used up by Google, or even Excite? What's the difference if we use it instead, and it possible works better for you in the end?
About the bootloader thing. Sorry about that guys, I didn't realize there was an Open Source project named Grub until we had the cards printed, domain registered, incorporated and had the plaque on the door. I've fielded a few emails about Grub (the bootloader) and we try to get them pointed in the right direction. BTW, we don't have anything to do with grub.com either
About the security problems. We have thought about this and do have a solution proposed (though not implemented). We are planning on scheduling the same URLs out to multiple clients, in much the same way that SETI@Home does. If we get bogus results back from a particular client, then we'll know fairly quickly that someone is pulling a fast one on us. There are a few other things we can do, but it will take time to implement them.
About the database. We really don't know about licensing the thing. Any comments or suggestions are MORE than welcome. We would like to leave it open for anyone to use or query, but charge LARGE corporations (like Google) for accessing LARGE bits of it.
Give us some ideas on what you would like to see us do and we'll listen.
Kord
kord@grub.org
Oh fun (Score:1)
--
Re:What's the license on the database? (Score:2)
- Q: What exactly does grub.org do?
So there are supposed to be selfish reasons for people to run grub nodes.A: grub.org is a company with a single purpose -
...We will make all software written during the project Open Source as well as all the hows and whats of setting up the network and database. If there is someone we can help by sharing what we've got, we'll share it.
Q: That's insane, what will Grub's revenue be if it doesn't charge for the software?
A: Open Source is not synonymous with NOT making money! We have come up with a hybrid business model that uses four distinct methods for generating revenue...
By placing the crawler closer to the data (i.e. on the web server itself) our client will be able to analyze and index the data local to the system on which it is running.
Q: So if I were a system admin or a website author I'd want to run the client?
A: Yes! Anyone that provides web hosting/authoring services will have a use for running our client. In addition to crawling a portion of the Internet, the client can index the admin's/author's entire site each and every night, and then submit that summary to grub's servers for incorporation into the database. Running the client will allow them to provide an added value for their clients - having their web pages updated to the biggest index, each and every day.
--
I wonder (Score:2)
What do you guys think?
------
Re:What's the license on the database? (Score:2)
As an investor, I wouldn't look to favourably on a business plan that relied on keeping the great unwashed slashdot audience happy (who else is going to run the client?)
Re:Problems with Google (Score:2)
Words like "to", "that" and "the" are known as Stopwords in the search engine biz. They are not indexed because in and of themselves they contain no valuable content. They are only valuable as part of a search phrase where they lend some additional meaning to the content around them. No search engine will include these stopwords in its index. Google would not be able to survive on a mere 4000 machines, it would require 20,000 or so were it to index stopwords as well.
When you do a search for a phrase - indicating so however that search engine prefers the syntax,usually just by putting it in quotes - the search engine does look for placement usually, even if it ignores the stopwords. So the number of words between two indexed keywords should be considered when it is searching the index.
As for the way the interface works - it differs from search engine to search engine, but I would agree that it ought to support simple Booleans and probably phrasing in the manner of mathematical statements using brackets.
These guys don't have to compete with google (Score:1)
don't really need to compete with google's search heuristics. They'll
be providing a genuinely new service that could comfortably coexist with
google's excellent searches of static pages.
Report dead links (Score:1)
Re:Problems with Google (Score:2)
You should read the Google help page [google.com]. But here are some hints:
I'm not big on reading instructions either, but there are times when a moment's research pays big dividends.
Re:_I_ don't want to lose Google's _cache_ (Score:1)
Re:What's the license on the database? (Score:1)
This build has been compiled and run on:
Linux RedHat 6.1 (2.2.12-20)
Linux RedHat 7.0 (2.2.16-22)
can't run it on windows, bwahahahaha
Re:No winblows client (Score:1)
GrubNote! (Score:1)
I wonder how we prevent the GraceNote scenario from occurring again?
Abuse issues (Score:1)
crawler technology... (Score:1)
i.e. why search site unless you know it's been updated.
Noble effort, comments? Let them know your stance. (Score:2)
Not to begruge them... (Score:2)
While they initially might seem big and evil, the source to their product is freely available [sourceforge.net].
Is it a crime to make money? They haven't hid anything. They clearly spell out what they plan to do with the data that they collect and how they plan to make money.
How many open source projects, while being really good projects and open and free (in all senses) have lasted? Eazel being the most recent example. If they use open source and make money this can only be a good thing.
If you don't like it, download the source and do it yourself.
JM
That's what I call multifunctional! (Score:1)
Well, as long as it boots Linux I don't care what it REALLY is...
Someone should pull a freecddb... (Score:2)
However, since their web crawling client is under the GPL, wouldn't it be possible for someone or a group of people to start a free (as in freedom) project using the client and a database that will always be available to the public? This would be a great idea, and would get many more users than a company that is likely to just close off its database after thousands of volunteers made contributions.
Because, after all, that's the real power of Free Software.
--
Sorry, I'm all volunteered out... (Score:2)
Re:Why bother? (Score:1)
My fundamental issue is with the following statement from your investors sections:
"...on what placement they get in Grub's search result sets."
This would imply to me that if I pay money to have my site indexed regularly, my links will be placed higher in priority over other similar links. Yes, many commercial services do this. However, they don't rely on community bandwidth to do their indexing.
The benefit your service would provide to the average user would more reliable searches as the index data is up to date. Mucking with the relevancy of returned data based on corporate funding changes this - in my opinion it makes it much less worthwhile. Hence much less worthwhile for me to provide valuable resources for indexing.
Now, if you were to provide a benefit to the users of your indexing engine other than searching then it would be a different story (perhaps pay them for their services?).
Perhaps we need a mojonation of the search world. If I run a search agent, then I earn mojo/points/dollars/whatever. I can use these later to get faster searches. If I don't provide cpu time/net bandwidth, then I need to provide some other form of input (such as money).
Why bother? (Score:4)
Let me get this straight: they want me to run their client on my machines, using up my cpu and network bandwidth so that they can resell that information to other search engines?
I particularly like this piece from their "Investors" page:
Third, Grub will begin charging website customers for content control. Content control consists of indexing updated information on a regular basis and controlling link placement in search results. Large sites who's revenue depends on sustained inbound web traffic will be charged based on the amount of data that they submit into Grub's database, and on what placement they get in Grub's search result sets.
So basically, the sites who are will to spend the most money will get their url's pushed up to the top of the list. Relevancy be damned.
Someone please tell me why I should dedicate my resources to this?
I think the smartest thing about the whole idea was putting the whole thing under the guise of an "open source", "peer to peer", "distributed", "let's make the world better" search engine. They might have managed to get some real interest if they had done a better job at hiding their financial motives.
Re:Problems with Google (Score:1)
That's the least of your problems.
Lastly, it's still not clear to me whether a search for "naked cheerleader" gives the same result as "naked cheerleaders". Hence, I tend to use OR and AND (+) a lot in my searches, which as I just said doesn't seem to work very well.
Here's your real problem. How can you tell that they're cheerleaders if they're naked?
not many users.. (Score:1)
Re:Problems with Google (Score:2)
What will happend to Google now? (Score:1)
"With a rubber duck, one's never alone."
I sense a proverb. (Score:1)
Build a better search engine and the world will beat a path to your website?'
--Jo Hunter
Neat idea - but I'm gonna pass... (Score:5)
Thank goodness for Google!
But again - this brings up the question similar to what happened with CDDB. Here you have internet volunteers providing free CPU power and bandwidth to provide raw material to for profit companies. Now granted - it is slightly different since you can still Google for free :) I'm not that selfish, but obviously there are some companies I'd be HAPPY to play a small part in improving their data set (Google) and others that given recent developments with URL submission and monetary sorting of search results that I wouldn't want to give data to unless they paid for it :)
Which now that I read the site more is their business plan. Read their Investor Page [grub.org] I get a squirrely feeling about this. I don't care if the client is open source or not. Why should I use up my precious bandwidth to supply content to a for profit company to sell to other for profit companies? Yes, they give the data away to non profits, but heck - most of them use Google anyway :)
And of course they are following hte lead of the other greedy search sites - adjusting search result order for money which I can't stand. Google is the one search engine that got it right - sort data by relevance and popularity.
I'll read more about it - but I think I'm gonna pass on this on - I just don't see the benefit for the volunteers who run this both on a selfish individual scale and a broader Internet community scale
--
Re:What's the license on the database? (Score:5)
Here is the section outlining what they plan to do with all this free data 'volunteers' give them:
The first revenue stream will come from selling URL status information to companies like Google and Altavista. This status information will enable existing crawlers to target the crawls for a particular day, based on the highly up-to-date information contained in our database. These status updates are similar in nature to the service provided by someone like NetMind, in which a change on a website triggers an action. Grub's database will be much vaster by comparison however, enabling it to provide services directly to wholesale search engines.
Second, Grub will begin selling "wholesale searches" to other search engines and companies. Grub will make strategic alliances with other search engines much in the same way that Google has done with Yahoo and Inktomi has done with Hotbot. Grub will also provide one-shot search results for a large search query, delivering the data in a database format (like XML) instead of a web format.
Third, Grub will begin charging website customers for content control. Content control consists of indexing updated information on a regular basis and controlling link placement in search results. Large sites who's revenue depends on sustained inbound web traffic will be charged based on the amount of data that they submit into Grub's database, and on what placement they get in Grub's search result sets.
Fourth, Grub will provide consulting services for companies wanting to set up their own Grub networks. Large corporate intranets could be quickly and efficiently indexed into a central database with the Grub client/server model. Consulting and coding for these proprietary installations is a common model in Open Source oriented businesses like Sendmail, MySQL and Apache.
Guess they thought we were really that stupid!
--
They say they WILL provide a searchable index... (Score:5)
Q: How much longer will it be before grub.org has a searchable index?
A: The first phase of the client and server project has just started. We expect that phase to take somewhere between 2-3 months to complete. At that time, we will begin deploying to the client to beta testers - at which time the database will begin to grow. A searchable index will become available sometime between now and then that will access the database directly. Update: We expect the database to come online sometime in Jan 2001.
Looks like they are a little behind schedule on this one.
And another telling tidbit from their FAQ:
Q: Why would I want to run this client? At least with SETI, I'm doing something - like looking for aliens.
A: We like aliens too, but ours is noble cause if there ever was one - to have a decent index of the Internet free for any individual to use when they need it. The reasons that you'll want to run it will vary, but we think you'll see the advantages to be gained by running our client - especially if you are a system admin, or author of a web site.
So I guess my main concern is that a) they could pull a GraceNote and b) the whole selling top result spots to big companies that may have NOTHING to do with what I'm searching for.
--
Re:What about manipulation? (Score:2)
This is something web masters could do to existing search engines like Google as well. From a technical point of view, a search engine's crawler, or any other client, does not request pages from a server, but invoke methods of objects which are named by URLs. So to fool search engines one simply has to make sure the GET method of such an object returns different results to crawlers and browsers. In addition the web master would have to hide the fact that invocation of the GET method returns dynamically created content, but that's simple.
Why not free search in the client? (Score:1)
Re:You've just taken the bait... (Score:1)
"Large sites who's revenue depends on sustained inbound web traffic will be charged based on the amount of data that they submit into Grub's database, and on what placement they get in Grub's search result sets."
The freeness of the client code is not the problem here, it's how they're planning to implement the search engine based on data that they receive. And ofcourse the client code has to be open souce to get wind under it's wings, but to use the good impression the mere words "open source" give to promote their data selling/mangling schemes strikes a bit odd to me. Atleast it's the first such effort I've seen...
--
You've just taken the bait... (Score:2)
As far as I can remember this is pretty much the first real business effort that's really trying to ride the "open source" reputation.
Even their web site front page has
"it's an open source, distributed internet crawler!"
What a load of crap, all I see is dollar signs are flashing in their eyes when they're just hoping to score big by selling those slots search result slots, open source or not.
A while ago it was multimedia this and interactive that, then it was everything starting with 'e' and now it seems to be the pure greed in "open source" / "GPL" disguise.
I can't wait for M$ to release their Open Source GPL'd office200x with some copyrighted document formats for what you need to pay $1000 licence fees... I hope atleast then
--
Re:What's the license on the database? (Score:3)
We could use the existing Grub software, but modify it to report to a community-run free database. Modifying the software to report to different servers would be rather easy, as would reverse engineering and replicating their database and website. However the sticking point would be funding for the large amount of bandwidth a site like this would need.
Problems with Google (Score:3)
That said, I get frustrated by some of its quirks.
For one thing, it excludes common small words like "to" "that" "the". Those words can be important when you're searching for a specific quote, say an old song or a line from a movie that you once heard.
Google doesn't seem to understand strict, logical use of parentheses, almost like it's really searching for the characters "(...)", or even the word OR, which contradicts my first complaint!
Lastly, it's still not clear to me whether a search for "naked cheerleader" gives the same result as "naked cheerleaders". Hence, I tend to use OR and AND (+) a lot in my searches, which as I just said doesn't seem to work very well.
No winblows client (Score:1)
Re:What's the license on the database? (Score:1)
The GPL protects from this (Score:1)
There is no risk of the system becoming proprietary, because it is truly free. There is also nothing morally wrong with making money using free software. It is only bad when money becomes such a priority that the information is made proprietary. The only risk is that they may license the data in a different and non-free way. So long as they use the GPL and collect their data under a Free license, there is no risk.
An admirable project! (Score:2)
This project is entirely free. Thus it is much better. People should go to the project homepage on sourceforge [sourceforge.net] and help out. The current goal is only to index content, and a later stage will implement intelligent search functionality. See the project overview here [grub.org]. I am sure that they would love to have more people who are able to do that helping out.
Hackers, get involved with this project that can replace one of the most used pieces of proprietary software, the Internet search engine!
Re:I don't think they can use the name "grub" (Score:1)