Wikipedia Founder Working on User-Powered Search

An anonymous reader writes "Jimmy Wales, founder of the Wikia corporation, has revealed plans to offer a user-driven search engine. Ars Technica reports that the plan is to leverage user preferences to pick the 'best' site for any given search term, while at the same time utilizing advertising for commercial gain. The article admits this may not be the ideal solution: 'Users may be reluctant to contribute to the betterment of a commercial site that may end up being bought by a bigger company. Consider, for example, the tragic death of TV Tome, a comprehensive community-driven television content guide that was eventually bought by CNET and transformed into a garish, excessively commercialized Web 2.0 monstrosity of significantly less value to users.' Just the same, Wales seems very enthusiastic in the Times Online article highlighting this venture."

Time will tell who is right but

[atw]

Disclaimer: I am a founder of a competing search engine concept that is based on volunteers running distributed software that crawls pages, conducts partial analysis and indexing before passing over results to central server that will index data into main central index: human aspect here is the people who take part in the project, and participants can actually change ranking formulaes, and shortly will be able to assist in human detection of spam etc.

Searching the Web is a very challenging problem (that's why few companies do it): volume of data is huge and one only appreciates value of good algorithms when faced with situation when poor algorithms make stuff run for weeks failing near the end and you have to restart the run to wait another week. You can either try to handle this very big problem, which is very hard even if you have the money (look at Amazon's A9 funded with millions, yet they licensed Google's code and database), or you can try to reduce the problem: only focus on a handful of "important" pages - Yahoo did that when they were human edited directory/search engine hybrid.

It seems to me that Mr Wales entertains the illusion that a very small number of manually checked pages in the Web space will be sufficient to satisfy vast majority (and it has got to be 98%+ as I won't be hopping from one search engine to another) of search queries. If this was the case then we would still be using Yahoo that did pretty much just that, yet almost everyone (including Yahoo) moved to algorithmic search engines because it is the only way to handle billions of pages, and billions of pages you will have to handle: even if you just index homepages of all registered domain names you will be dealing with 100 mln+ pages, that's good 20 times more than articles in Wikipedia and checking pages can be far more duller than reading nice article you have some personal interest in.

What I find ironic that our own concept of the search engine was removed from Wikipedia because we were supposedly "not noteable enough", that's the sign how they handle problem of "too much data" in Wikipedia - they just reduce the problem by reducing datasets greatly, sometimes this is done wrongly, sometimes rightly and it might well work for Wikipedia, but it sure as hell won't work for Web scale searches. Oh, and by the way who said Google and others don't use human reviewers? They sure do, just check TrustRank [wikipedia.org], this link is ranked as #1 match on Google for search TrustRank! Notice what Wikipedia tells us: "While human experts can easily identify spam, it is too expensive to evaluate manually a large number of pages."

Human input plays an important (although fairly unknown as they prefer to keep it secret) role in the state of the art search engines, however suggestion that humans can handle billions of pages and/or that a handful of pages will be sufficient for a general purpose search engine is wrong and a very backwards move that will result in exactly the kind of wrong attitude present in Wikipedia now.

Doomed to repeat it, I guess

[hirschma]

OK, perhaps the new generation of Web entrepreneurs had better learn something from their "elders". We're about to see a lot of concepts that didn't work in the 90's be resurrected and funded. This has been tried and failed, badly:

An orphaned ref to Magellan, the human powered search engine [gocee.com]

Didn't work before when there were a lot less sites out there, not likely to work this time, either.

jh

A new social search is already out there.

[KrissyM]

SearchSays.com [searchsays.com] is a user-based search engine that just recently launched, and has a growing contributor base. Looks like it might be a race, but I guess I'm not surprised it's already been started. There are so few original ideas on the Internet these days. As to the "it's already been done and failed" remarks - timing is everything isn't it? I could see something like this taking off with the current Web 2.0 craze.

Actually this is not going to happen

[GerardM]

Hoi,
On a WMF mailing list Angela said that there was no substance to all this. I had also heard from other channels that there is not much to this.

So even though it is nice to speculate, there is not much to all this.

Thanks,
        GerardM

After getting burned with CDDB, forget it

[John Jorsett]

I entered a bunch of my CDs into the music CD database CDDB [wikipedia.org] (now Gracenote), thinking that the database's contents would remain public domain or at least freely usable. Then it was sold and the company that owned it forbade access by media applications without a fee. Thousands of people like me voluntarily built that thing, and now they wanted to sell it back to us. Any organization asking for my help on future projects had better have an ironclad guarantee that my work product will remain free to users.