Nutch: An Open Source Search Engine

Anonymous Coward writes "Someone forwarded me this site working to create an open source search engine called Nutch. In the age of weighted rankings on search engines for profits, there's an obvious need for an unbiased search engine. After all, isn't a search engine supposed to be for finding relevant data, not as an indirect and sometimes slimy method of advertising? Nutch is clearly in their intial stages, but it would certainly get my vote." You can find the project on SF.net, and also read the Business 2.0 article on it.

Re:Hook it up to slashdot!

167 posts and no mention of ht://dig [htdig.org]? It's a great open source search engine, and I've been using it daily (well, cron really uses it now, not me) to spider about 100 sites on my intranet, which has servers all over the world.

While not currently designed for massive whole-web spidering (it's aimed at single websites or intranets), ht://dig is a great starting point (and a lot further along than the Nutch 'nascent effort' mentioned in the story). Some database optimization to ht://dig seems easier than starting over with Nutch. Plus, the name 'Nutch' sucks.

Re:Hook it up to slashdot!

Interestingly enough, if i had read this story a few months ago, i would've said "Poppycock! Google should be good enough for anyone!". But lately i've been noticing that Google turns up a lot of garbage results. Like, if you search for something "generic" (like, no brand name or product name or anything like that), you're going to find a whole bunch of results that just lead to pop-up search sites.

For example, look at the results [google.com] for the search 'convert wmv mpeg'. The first three results lead to the same exact search site. (Whether they have pop-ups or not, i can't tell, because i block them.) The fourth result is another search site. And then the last three are the same as the first three.

Of course, this obviously works with stuff you'd expect it to, like 'mp3s' and 'warez' and 'porn', but it works with legitimate stuff too. I wonder if there'll be anything to combat this trend, whether it be implemented by Google or by someone else....

Patents.

I hope the authours of this project do their homework. My impression is that most of the good search and indexing schemes have already been patented, which will make it difficult to release such a project without stepping on someone's toes.

Re:Lucene (index and search engine)

Lucene and Nutch are related:

http://scriptingnews.userland.com/2003/08/13#When: 12:20:53PM [userland.com]

Paul Nakada, via email: "It appears that the coding muscle for Nutch is Doug Cutting, the author of Lucene, an Apache Project open source search engine. We use it here at salesforce and have a huge amount of respect for Doug's coding."

Re:Patents.

I hope the authours of this project do their homework. My impression is that most of the good search and indexing schemes have already been patented, which will make it difficult to release such a project without stepping on someone's toes.

Hmmm, I just realized something... with patents, you end up stepping on people's toes. Without patents, you get to stand on their shoulders. Which do you think is the better vantage point?

Re:Patents.

Does it matter? There are no innovations. ALL knowledge is based on prior knowlegde. Look in any field of study and you will soon learn that advancement is not possible without prior knowledge. What we know about computer science today is thanks to the knowledge gained by those before us. It is this way in EVERY field, Astronomy, Medical Science, Mathmatics, etc. Humankind does not grow by leaps and bounds, we grow by incremental improvements. I have not heard of ONE discovery/innovation in which the discovery/innovator was not educated in prior knowledge. Now the question we need to ask ourselves, and especially the government is do we really want the advancement of our society to be hindered by monetary interests of the greedy?

Re:Patents.

In practice you may be right, but the intent of patents is the reverse. The key thing to think about is that without patents there is an incentive to keep ideas secret. So, you end up standing *beside* people until the idea comes out. If something gets patented, it is public knowledge, and you can stand on the person's shoulders so long as you pay them a "small" fee. Even without their consent you can do research that takes advantage of the knowledge in the patent.

Of course, in practice patents are a mess. ;-)

Re:The purpose of a search engine

Here you go.

Porn [sublimedirectory.com]

Anti-Microsoft Propoganda. [slashdot.org]

Google?

Last i heard google still doesn't accept bribes for page ranking.

inobtrusive adverts on the right hand column nonwithstanding.

Re:Google?

I have to agree. And I don't see my allegiance to Google as a sell-out. I see it as a reward for good work.

Slimey adverts?

Yes, having advertising affecting search results is not good for the end user but (and I'm just bringing this up as a discussion topic), in what other ways can a search engine make money? It's clear that running a search engine has costs associated with it. To offset these costs, it seems like advertising is the only way to go. Now I can see that some search engines handle this in a more "slimey" way than others (I am happy with Google) but this project seems to want to avoid advertising at all costs. Where does the money come from then?

Also of note is that companies can still influence search engines in slimey ways - Google can be manipulated to make a page rank higher, although Google keeps an eye on this activity and works around it.

Re:Slimey adverts?

This project is the SOFTWARE to run a search engine. Not a corporation that needs to generate income to justify the resources required to run the search engine.

Anyone could take this source code and with enough money, challenge Google.com as the top search engine.

I see this project as a competitor to shrink wrapped search engines. IE google appliance [google.ca] or maybe even Folio based products. Typically corporations have many documents that need to be indexed and searchable to their needs.

I haven't seen this on the homepage but it doesn't list what content it can index. I hope it can at least index PDF's and popular Office documents.. Maybe even Media files? And what XML indexed fields? Or external metadata?

Shameless plug for SWISH++

I see this project as a competitor to shrink wrapped search engines. IE google appliance or maybe even Folio based products. Typically corporations have many documents that need to be indexed and searchable to their needs.

SWISH++ [mac.com] fills this niche nicely. It can index hundreds of thousands of documents very quickly, indexes not only HTML, but e-mail, news, man pages, LaTeX, RTF, and even the ID3 tags of MP3 files; can apply filters on-the-fly (convert PDF to text, then index that), can do incremental indexing, and can run as a multi-threaded search daemon.

Biased listings

I think many commercial search engines have learned that biasing themselves to sites who have paid them is a good way to errode consumer confidence, and damage their readership/userbase. Just as newspapers have to at least provide the image of objectivity, the same demands are on search engines.

I'm quite comfortable with how Google does this (present commercial links clearly marked to the side), and am not convinced a non-commercial (open source) alternative is needed.

just don't get it

I think that you absolutely have to have a closed source algorithm for ranking pages, because otherwise you'll get people who will simply tune their pages to be high on the list. I can see how making the majority of the search engine open source would be beneficial, but the algorithm itself? Its like saying "Here's the keys to my car" and thinking that, because everyone has access to the keys, no one's going to drive away with it. Sure, everyone has the opportunity to make your search engine better, but never underestimate the tenacity of a web-wanna-be-millionaire.

Re:just don't get it

Think about cryptosystems: The whole point about the really good ones is that you can know the algorithm, but still not break it. Granted, pulling that off for a search engine is prone to be much, much harder -- but I *do* believe it's well within the realm of possibility. Ambitious in the extreme? Certainly... but there's something to be said for high-risk-high-reward projects.

Search engine game is NOT over

"Google has WON the search engine war, probably forever. Find some other mountain to climb, guys."

At one time, Oldsmobile won the auto company wars. Where are they now?

IBM ruled the PC roost. Hmmmm....

Command-line OS's were king. But now???

Altavista and infoseek and Lycos were search engine kings at one time. Whither this trio?

The point is, it is not over.

Accuracy is relevance

To me, accuracy is the most important "Relevance".

The problem with Google is that there are errors in it: you ask for something and sometimes you get something else.

A search on "to be or not to be" produces an error (non-matching results) in three of the first ten results: a 30% search failure rate. It used to be worse, when most of the links were bad.

Since it seems like Google will never fix this problem, I'm looking forward to something with all of Google's great features, plus accuracy.

Re:Accuracy is relevance

A search on "to be or not to be" produces an error (non-matching results) in three of the first ten results: a 30% search failure rate. It used to be worse, when most of the links were bad.

This is a bit of a misrepesentation. Google will toss the words 'to' 'be' and 'or'. So you effectively end up searching on 'not'. It does this to eliminate words that show up to frequently and make the searches faster (and the overloading of the word 'or'). If you really want that text, then either quote the whole thing, or place a '+' in front of those words, which will give you exactly what you're looking for. So there is no problem with it's acurracy when you understand the proper way to ask it for something.

Re:Accuracy is relevance

If you reach into the freezer without really looking, thinking that you are grabbing a freezer-pop, and get an 8 month old leg of lamb instead, are you going to shrug and eat the lamb anyway?

Of course not. I'd put it back and try more carefully to get what I want. I, what's the word I'm looking for, . . . wait for it . . . refine my search :)

Regarding your comments above about google inaccuracy: I searched for +"to be or not to be" [google.com] and consider the first page of 10 hits to definitely be 100% "correct". In fact, all of the 104,00 results that I checked (about 50, hehe) are 100% correct in that the sites on the list, or the sites linking to the sites on the list, contain the phrase "to be or not to be". Check the '2bee or nottoobee' link in google's cache and where you normally see the search term highlight colors, you'll see

These terms only appear in links pointing to this page: to be or not to be

Just because you wanted "Shakespeare" doesn't mean that "Shakespeare" is any more correct as an "answer" to "to be or not to be". If it were more popular (on the web), I'm confident that it would be higher on the list. That is, whether we like it or not, on the current www there are exactly 3 things more relevant to that famous phrase than Shakespeare, and they are, in order: barium enemas, beOS, and a kids' grammar game starring a bee. Or, more acurately and revealingly: an article about barium enemas titled "To BE or Not to BE?", an article about BeOS titled "TO Be OR NOT TO be?", and a kids' grammar game starring a bee called "2Bee or Nottoobee" which is linked to by sites containing the phrase "to be or not to be" in or near those links.

Lucky for us that ol' Bill is still in the top 10 at all, I'd say.

Seems pretty pointless

Free and open code is good and all... but the one real cost of a search engine is RUNNING it. It requires a far from trivial amount bandwidth and hardware, and somebody has to pay for all of it. Unless someone comes up with a novel P2P solution (and many are trying) it just won't happen.

What they should be doing is pressuring the existing search engine companies for some integrity.

not a good idea....

google is already ideal... the weight of search results is not sold, just text ads.

people are already 'googlebombing' to try and get better rankings by signing up tons of domains and cross linking them all with the keyword that they want to be #1...

if the algorithm that determined how #1 is determined was public, then the best possible strategy to cheat the system could be demised... instead of paying for weight to the search engines you would be paying to web developers to make the search engine think you were #1. and as a web developer i feel that.... oh... wait, proceed.

Can this work?

I think the idea is good in principle, but could it actually succeed? Google gets hit with millions of request each day. They've got hardware that can support thousands of slashdottings a day and a fat pipe to feed all of that info out. That takes alot of money. Financing an open source project is difficult enough, but financing an open source service such as that would seem next to impossible. Ideas?

The other major problem would be that, with the ranking criteria being available for all to see, it would be relatively simple to manipulate page rankings.

A Tough Challenge

One of the biggest issues with running a search-engine, open-source or otherwise, is that you can't eliminate bias in the results. No matter what scheme you put in place to handle rankings, someone will find a way to take advantage of it. It's a fact of any major system - there's always a way to twist it. Part of the challenge that Google and similar sites face is that they have to work constantly to protect themselves from systems designed to take advantage of their algorithm. While a completely unbiased search service would be nice, I think it would require the impossible. It would require that no one out here took advantage of it to further their own interests, be they political, commercial, or otherwise. That's fairly unlikely.

With most of the major engines today including Google, they make an effort to prevent horribly unbalanced results (recent controversy over blogs outweighing professional sites in the rankings due to linking and other factors). Some even admit (again, Google does) to manually messing with the rankings a little. If you search for suicide methods, they will bend the engine to make sure you get reasons why you shouldn't commit suicide before you get the how-to. That's in their own public docs. It's also discussed in Wired.

I honestly don't know if open-source could do a better job. The algorithm might be better (likely, given the manpower), but would it really be that much fairer?

Are they thinking too big?

I think they're setting themselves up for something that will get too big and too expensive before it can get finished, and they'll have to figure out a way to (gasp) get some funding beyond donations.

I don't see a solution in one great open-source, independent search engine, but many individual specialized search engines, each mastering their own niche area of specialty stands a chance to compete, especially if run by people who focus on their areas of expertise. Alternative news search engines, music search engines, literary search engines, etc. each run by people who know what to filter in and out.

If Nutch.org could create the technology that would allow each of these search engines to exist autonomously, it could also be the hub/portal/start-page/blahblahblah that links all these engines and databases together.

Alex.

The answer is "Nutch"

*Blows open envelope*

The answer: "What did Sean Connery say when he saw the reviews for 'League of Extraordinary Gentlemen?"

Let's check out the credits page...

Ooh, what's this?

Overture Research has donated hardware and helped to fund development.

So, even an "open source," "unbiased" search engine is funded by a commercial search organization.

funding

let's see where is the funding coming from. Project is funded by overture [overture.com] which is to be bought by Yahoo [yahoo.com]. More info is here [corporate-ir.net]. Hmm.. So i guess Yahoo needs a revival...

Distributing the Power

I think having an open source search engine that people can modify and deploy would be an excellent thing, and here is why. Currently, google has the complete power to highlight or censor anything on the web. So far, they have used this power wisely, but that's no guarantee that it'll always be so. If they go public, you may find this power being used to increase the shareholders' wealth, rather than in the highest standards of fairness as it is today.

With that in mind, how would this project help? It would allow webmasters to quickly & easily modify it for their needs, and deploy their own niche engines; in other words, Google would be supplemented by 10,000 niche search engines, each focusing on a specific field (microsoft propaganda, for instance). This would create a balance of power, ensuring that no single search engine accumulates an insane amount of control over the web as a whole.

Bias: Inevitable

"In the age of weighted rankings on search engines for profits, there's an obvious need for an unbiased search engine."

Bias is inevitable -- we're talking about ranking, which necessarily means bias.

The question is: what bias do you want? What bias suits your purposes?

My ideal search engine would offer a variety of biases from which to pick.

Search Engine Monoculture

Why is it that when it comes to OS, everyone is bitching and screaming how bad monoculture created by Microsoft Windows is, but otherwise feeling warm and fuzzy and swear to god Google is and always be the only search engine they use?

The point is, are you really comfortable to have one, and only one, effective search engine? No matter how well it searches?

O'Reilly [userland.com] put it best :

Actually, Nutch has no ambitions to dethrone Google. It's just trying to provide an open source reference implementation of search to help keep Google and other search engines honest, by letting people compare the results of an engine whose algorithms and methodologies are transparent and accessible. It also aims to give a platform for people outside of the search heavyweights to research new search algorithms.

Irrational fear of money

That's nice that they want to open source the engine but that's the least of a search engine. They're going to need multiple high end servers to process the searches and plenty of bandwidth to get the results to the users.

How do they plan to pay for that? Apparently advertising is out. And we just had another monephobe complaining about lack of funds for his accounting software who expected people to donate because he couldn't figure out that maybe, just maybe he should find a way to sell his product in some form while also keeping one form free. I can get RedHat for free OR pay money to get a hard copy with some bonus stuff. Net result is that RedHat makes money and everyone is happy. Those who refuse to pay don't have to and those who are willing to pay have a reason to. Most people are not going to just give you money out of the goodness of their heart and accept nothing in return if they don't have to. Why do you think PBS gives you gifts with your donations?

I'd be more impressed with such undertakings if the owners weren't convinced the bandwidth fairy was real and that money will fall from the sky like mana.

When someone comes along who recognizes that the bandwidth fairy doesn't exist and that money needs to be aquired through marketing to get any real amount then I'll think twice before laughing it off.

Free is a pretty dream but free don't pay the bills.

Ben

Large scale and DB

I was looking over the site and a number of things concerned me.

Firstly the choice of Java, personally I have no gripe about this. And reading that a choice was made to use language-independent formats is a good idea. My main concern is for the larger scaling and distribution over multiple machines.
At present I make the educated guess that a project on this scale, in Java, would still be best run on a `hardware base as uniform as possible', like UltraSparc 450's with a fibre back-plain.

My second concern is that there is so much choice of indexing and searching technique that there are sure to be some problem due to Patent restrictions.
Just browsing the US patent office gave me a couple of possible Patent nasties;
6,463,428 or 6,278,992. (And about 10 others I glanced at...)

Lastly DB, in the short time I've been looking at the code it seems to me that a choice was made to implement a DB build for the problem. Although this could be a good thing, it is usually better to reuse existing products. I found SleepyCat (DB4) to match the requirements. And if the choice is final read this. [1]

I hope these comments are useful to somebody at least.

[1] http://www.xlnt-software.com/xml_dl.html

Some commentary...

I have a few comments on this development:

• The article as posted contains some pretty snide commentary, apparently designed to intimate that all current search engines deliberately weight their results in favour of their advertisers. This is demonstrably not the case; in fact, with Google providing a strong, well-publicised counterexample, to do so would be suicide for any search engine with pretentions to market leadership.
• The principal difficulty with an open-source search engine algorithm is that it would definitively be open to abuse. Once the ranking algorithm was known, it would be fairly trivial to develop ways to subvert it. One of the reasons why this hasn't happened to Google is because the details of the ranking algorithm are closed. There is a largish industry devoted to figuring out how to influence Google (which is why Google keep tweaking their algorithm). A search engine using an open algorithm would very quickly become unusable as this industry figured out how to play the system.
• The funding from Overture is very suspicious, to be honest. Overture, assuming the Yahoo! takeover is given the all-clear, will soon be part of one of the largest commercial search engines, and with a history of business practices which are, shall we say, perhaps less than totally congruent with the open-source ideals.
• Running a large, successful search engine requires vast, dedicated resources. I don't know the exact scale of the Yahoo!, Google or MSN search operations, but I'll warrant that they're surprising to anyone who's expecting to run a search engine from a couple of thousand distributed nodes.

An open search engine application is a nice idea, but unfortunately it's one of those applications which are essentially useless without an enormous ASP architecture behind it. An earlier poster indicated that it might be useful for searching and indexing intranets and the like, analogously to the Google Search Appliance. This is indeed a valid potential application, but then, HT://Dig exists already. Is this dramatically better?

Comments and suggestions...

Suppose you have just finished developing a free software search engine. And suppose it has the best algorithms in the world and the ratings are weighted based on some sort of moderation system.

This is exactly like the problem the mice had one day. They couldn't come out of their mouse hole because there was a dangerous cat prowling around. One day, as food was getting scarce and everyone was afraid to leave the hole, the mice called a meeting to discuss the problem. One excited young mouse came up with the most wonderful idea: Let's put a bell around the cat's neck, so that when the cat is nearby, the mice would have advance warning and could escape! All the mice got excited at this proposal, until a very old, very wise mouse came over and asked, "And who will tie the bell around the cat's neck?"

What I'm trying to say is: If the search engine is free software and companies don't pay to increase their ranking... who will pay for the bandwidth to host the engine? I can tell you this much:

• Individuals will not pay a fee to perform a search unless this search engine gives them some incredibly compelling reasons to do so. Open moderation will not likely fulfill that requirement.
• Companies will not pay to increase their ranking because that is the definition of this project. They will not pay to search for the same reason that individuals won't pay.
• The government probably won't pay because there are plenty of "free" (cost) search engine around. That is, unless someone can give them an incredibly compelling reason to do so.
• Universities probably won't pay for the same reasons as everyone else.

Proposed solution? Make it a distributed search engine, like SETI@home, or the DNS.

This is much easier said than done because:

1. RAID-like distributed storage technology would have to be developed, so that the indexing database could be distributed among all computers worldwide that donate bandwidth and storage. This would have to guarantee statistically that all the data will be available at any point in time even if people turn off their computers for extended periods of time. However, this technology could make reliable clustered storage a reality, and the resulting free software implementation could be licensed for corporate use for an exhorbitant price, which would go to the EFF, FSF and other organizations that develop free software and/or support the development thereof.
2. An efficient P2P-like protocol, along with a network topology of some sort (like the DNS system has) would have to be developed to support the searching; It would have to be damn fast and, like before, very resiliant to computers being shut off, chunks of data becoming lost at any moment, etc. Furthermore, changes would need to propogate at blazing speeds so that new items on the Internet could be found shortly after appearing.
3. Bandwidth and disk quota would need to be managed at each participating host, so that limits set by the user are not exceeded.

Governments, companies, universities and individuals would likely support an effort like this by donating some bandwidth and storage, rather than money.

In the spirit of worldwide computing on the Internet, I hope this makes some amount of sense.

Re:Not making nutch sense

Don't worry. It is just a stepping stone to full project maturity reached when it is fully coded in Borland Turbo Pascal.