Wikia Search Engine to be Launched on January 7th

cagnol writes "The Washington Post reports that Jimmy Wales, the founder of online encyclopedia Wikipedia, has announced the launch of a new open-source search engine, Wikia Search, on January 7th, 2008. The project will allow the community to help rank search results, in a model close to Wikipedia. However the company is a for-profit organization. This new search is supposed to challenge Google and Yahoo."

Challenging Google?

I guess that's their response to Google's Knol (http://en.wikipedia.org/wiki/Knol) Pity to see things heat up between the 'good guys'.

Re:Challenging Google?

No, it is no response to Knol. I have been working on this for a year. The press has talked about it endlessly. :-)

It'd be sort of cool if we could create a search engine in a week or two to respond to Knol, but actually it takes a bit longer. :)

I see Larry and Sergei socially from time to time. I spoke about the search project at Google Zeigeist a few months ago. Going to a google party next month. The media loves a "fight" but really, that's just a nice story arc the press makes up. (Notice: google is not in the search business, google is in the advertising matching business. This search engine doesn't hurt that business at all, indeed it probably makes it marginally less likely we will see the emergence of a proprietary competitor to topple them.)

It is actually possible for people to just enjoy doing cool stuff without being bastards about it. People forget this sometimes, maybe due to the reputation of a certain dominant software provider. :)

Re:Challenging Google?

It looks like you've entered some sort of partnership with Grub http://www.grub.org/ [grub.org].
If so, kudos... Grub's been languishing in not-ready-for-primetime land for far too long, and the ability to crawl your own site to keep results current is a bonus, too.

Re:Challenging Google?

No, it is no response to Knol. I have been working on this for a year.

I'm sorry, but your post cites primary sources and thus does not meet Wikipedia's standards.

Re:Challenging Google?

It is actually possible for people to just enjoy doing cool stuff without being bastards about it. People forget this sometimes, maybe due to the reputation of a certain dominant software provider. :)

Oh, come on. The people who matter already know that most Linux users aren't elitist snobs.

Re:Challenging Google's Revenue Model

From the RIAA threads we learn people don't want to pay as endusers for their content.

Great post, except this part doesn't make any sense. I pay as an end user for content all the time, and not just for high-end data: Magazine subscriptions, membership in various societies (and their publications), newspapers, my ISP, government funding (I pay through taxes), direct donations to non-profits, contributions to wikipedia and other open content systems directly. While some of them are for high-end data, a lot of it is not.

Is content going to ever be totally free? It will be if people understand the inherent rewards of an open society. Information's negligible cost of duplication is the revolutionary model is the thing that is shattering the old models (c.f. http://homes.eff.org/~barlow/EconomyOfIdeas.html [eff.org]). Wikipedia is already doing that. As much as I'm a critic of Jimmy Wales, citizendium, etc. (with their NPOV lunacy), the system he's helped build is saving people's lives and improving quality of life in ways the old world just doesn't understand yet.

Personally, I'm hopeful that as long as we still have the Right to Read (c.f. http://www.gnu.org/philosophy/right-to-read.html [gnu.org]), we're on the path to freedom and salvation. A corporation who makes up a new "model" to take advantage of content producers isn't going to take hold anymore. There's just not a point anymore. The price of content is already quite low for common knowledge. Even if the arbiters of knowledge try to keep it from common knowledge, we can paraphrase it. The greatest risk to real productive use of our knowledge still remains Patents. Information may finally be free, but the freedom to tinker is not.

Easily Abused?

So basically...they're asking for people to abuse the ranking system. To patrol something like this would require a company with resources like Google, and most likely the reason Google doesn't have such functionality. Just my two cents.

Re:Easily Abused?

What this means is that no matter what you search for, the top hundred results will be to porn sites.

Re:Easily Abused?

Having an open algorithm is good, as non-disclosure isn't security, but the issue is allowing people to rank searches and such. Having that public is asking for people to abuse the system, and as noted before, a lof of malicious parties could seemingly legitimately rank their sites (porn sites, etc) higher, leading to ranking battles by bots. Of course, the issue of vandalism occurs with Wikipedia, however when people are looking to make money off of it they'll likely be more persistent.

Re:Easily Abused?

Point well made - while spam attacks may be pretty obvious, they could be spread out over time to make them less obvious.

Additionally, I can see this search engine being very much affected by public mood. For example, say there was a royal death and a certain right-wing 'upmarket' tabloid newspaper [dailyexpress.co.uk] decided to claim that it was a conspiracy by the Government to kill the royal off. This is linked to from said newspaper's web site, and this people improve its ranking. Therefore it floats to the top of the results pile, thus giving it more exposure and setting off a vicious cycle.

Just a hypothetical situation, but certainly possible. Such a model would also make it possible to carry out smear attacks and to ruin the rankings of competing companies, parties, organisations, whatever - a practice that IMHO should be left to search engine admins.

Re:Easily Abused?

The question of abuse is obviously one that we are taking very seriously in thinking about design issues. My belief is that the key to solving this thorny question is hinted at by the success of wikis and the wiki model: the key is to put tools in the hands of the community that allow for broad oversight and control by the community in a process of open dialogue and discussion. This is very different from approaches that allow only for atomistic participation by a "community" which is never allowed to really become a community due to excessive reliance on algorithmic voting systems and similar.

One of the first lines of defense in the early days will be use of a community (wiki) generated whitelist [wikia.com] of sites to crawl. We will want to work outward from there, but basically the first thing is for us to assess "look, what are the most important must-have sites on the net" and crawl them. One thing that the mainstream media never seems to report very well, mostly because I think they don't get why it is important, is that we are doing everything here under free licenses. The software GPL, the data we generate under free licenses, etc. The aim here is not just to create a good search engine, but to create it and *give it all away* in a way that I think has a chance to restructure the entire search industry. Well, maybe not, maybe so, but what the hell, it'll be fun to see. :-)

Your track record says otherwise

Unfortunately for you your track record disagrees with your promises. You and your website have a history of abuse and bias that rivals that of any on the Internet. Your management incompetence of Wikipedia is so bad that you have dedicated websites documenting it. From secret mailing lists to the junior high style politics that rule your sham open organization, you are incompetent.

The thought that Jimmy Wales, cofounder of Wikipedia could have an open site without abuse is laughable. You operate under the sham of an open community, yet exclude those outside a very narrow political agenda. Your a fraud, using open source principals as a smokescreen that presents your personal world-view set as fact to the world. I don't buy what your selling, and I'm calling your bluff. The sad thing is that this will probably make you a fair amount of money if more people don't start to see through you.

But then the wonderful thing about leading revisionist history is you can substitute your own revisions for reality....

Re:Your track record says otherwise

Come to your wikipedia page?

you mean the one that you have been documented [wikipedia.org] (and here [wikitruth.info]) not only editing, but wiping clean the edit history on, trying to bury your tracks?

The game you're playing is dirty and how dare you come here unwilling to meet us on equal ground.

Re:Easily Abused?

Is there an intersection between the people who decide what goes on the whitelist, and what is "notable" for inclusion in Wikipedia?

I thought so. Your solution is already broken.

yeah

The Washington Post reports that Jimmy Wales, the founder of online encyclopedia Wikipedia, has announced the launch of a new open-source search engine, Wikia Search, on January 7th, 2008.

Not only that, Wikipedia is reporting that its marketshare has tripled in the last six months.

Market share of...

Market share of what?

Wikiality

Re:Market share of...

[[citation needed]]

I don't care how they arrive at a rank!

The idea is to challenge the established players by offering a search service that is more transparent to end users, meaning they can see how search results are arrived at. Wales has described Yahoo and Google as opaque services that don't explain how results are arrived at.

Personally, I don't care how search engines rank the websites they return as long as what is returned is proper, relevant and useful.

Vandalism

I predict significantly more vandalism and self promotion with this project than with Wikipedia. That said, I still think it's a good idea. But the article had a very low content:words ratio, so I don't really have a good idea as to how it will be implemented.

My prediction: killed by nonprofit competition

Since this project would seem to depend on the participation and good-will of users in order to work, my guess is that a nonprofit version will follow shortly afterwards, paralleling the open-source model. I also predict that without the benefit of a massive Microsoft-esque head start, the for-profit version will be put of business in short order.

What I always wanted

Someone loading the dice on what I get back from a search. At least with the current crop, I can more or less figure what they're doing. With a dynamic, anything goes approach, I seriously doubt I'll be using it much.

Biased Rankings?

I have a feeling the general user who would edit a wikipedia article or something like Wikisearch will be much more tech savvy and promote more tech oriented results for keywords than those that google will provide. This can be a good thing or bad thing depending on what you are looking for. Google personalized search is probably the superior method since it uses the users own searches(without external input) to determine the rankings of the search results.

I'm glad you told me

This new search is supposed to challenge Google and Yahoo.

Really? Is a search engine startup going to be competing in the same industry as other search engines?

first things first

It would have been nice to see them fix Wikipedia's own search engine, which IMO is absolute garbage. I have a better chance of being linked to what I'm looking for by using a general search engine.

Re:first things first

Search for Kobar Towers and you get 0 relevant articles. Search for Khobar Towers and you get 62 articles. Yeah, the first is a misspelling, but it's 1 letter off, nothing difficult for a spell checker to check against a dictionary of existing articles. What use is a search engine if it is so strict that I have to enter the terms exactly to get an article when I could just do that in the URL?

As long as I need to use google to search Wikipedia, I don't see Wikipedia creating a google killer.

Re:first things first

I completely agree. I am continually amazed at how good google's input-correction is - if I do a search for 'pale gire', it knows to correct it to 'pale fire [wikipedia.org]', yet if I do a search for 'canadian gire', it's clever enough to work out that I mean 'canadian tire [wikipedia.org]'. I'm also continually amazed that people running other search services haven't yet realised just how vital this feature is - it's probably one of my favourite things about Google. Less so for monosyllables, but it's useful for words like "monosyllables". I'm particulary surprised that prominent online dictionaries don't have similar funcionality, seeing as I would imagine a large portion of their usage is to find the correct spelling of words.

What a joke...

This is simply his response to Google starting what amounts to competition for wikpedia. I'm sure google is having quite the laugh from it - one wonders how much of the donations for wikpedia are being used towards this thing.

If you think wikpedia gets vandalized, wait until there's money involved. Wikpedia for all it's trappings, doesn't directly influence spam. But a search engine... IF, and this is a big IF, this thing becomes mainstream, having the code public will make it very easy for the bot herders to control it. The idea is simply flawed. Google is currently dealing with bot herders attempting to manipulate it's page ranks - while the idea of it being open source sounds great (well, ok it doesn't to me - I don't have the love affair with open source that most slashdotters do - I've never bought into the security myth that there's GOOD coders out there with so much free time on their hands that they are walking OTHER peoples code. I don't like doing that when I'm PAID to do it. Not too mention there just aren't that many good coders out there....but I digress) it's simply going to work right into the hands of the malware crowd - especially now that it's more organized crime than it is vandalism.

EK

Re:What a joke...

Again, it would be hard for this to be a response to Knol, since I announced it and have been working on it for a year. :-)

And, if you read the linked article, you would know that *zero* donations from Wikipedia have anything at all to do with this: Wikia is a completely separate organization.

Also don't make the classic mistake of thinking that "open source" automatically means "volunteer coders". It generally does not, and the classic FUD from the proprietary world fails to describe reality for precisely this reason.

And finally, one of the most important concepts here is that of a broad deep whitelist, which is something that I think can be done realiably and well with appropriate tools in the hands of the end users. The entire problem of bot-driven spam comes from a lack of reliable quantities of human oversight in the process. All you have to do to massively spam google is fool a computer. (Well, even then, google does a pretty damned good job of preventing massive spam though of course there are always some problems.) Pretty hard to get that nonsense by a properly organized community effort.

(But of course, the design of a community which can move things forward quickly without a lot of useless work is nontrivial.)

I can see...

by our tags, that we have a few Wikipedian Protestors [xkcd.com] in our midsts.

Just in Time for the Election

All right! Googlebombing is time consuming and an organizational nightmare. This will simplify everything. Karl Rove

Fix the wikipedia search!

Honestly, it's crap. It's like using infoseek back in 1995!

Google search for Wikia

"Results 1 - 10 of about 1,190,000 for wikia. (0.04 seconds)"

Someone had to do it ;-)

What a great idea!

But building a search engine is a little ambitious, even for the co-founder of Wikipedia. Maybe he should start off small, like searching one website. I even have the perfect one to start off with: its search feature is so bad that if your search is off by one letter, you have a good chance of not finding what you're looking for. Maybe you've heard of it [wikipedia.org].

Hope it works better than wikipedia's search

Shouldn't they work on getting wikipedia's search to work half way decently before they try to compete with Google?

Don't get me wrong, I like wikipedia, but their search on the site is next to worthless.

Well, good news

We know how the search engine will work, if anyone has ever used Wikipedia's search function, you're almost destined not to find what you're looking for, especially if you're one letter off. ;]

citation needed

The story lacks a link...

...hold on while I search for it on Google.

Sooo....

How is this different from Mahalo [mahalo.com]? The media wiki powered search engine that has been up for almost a year?

Re:Sooo....

Completely different. :) For one thing, we are doing everything completely freely licensed. Mahalo is proprietary.

For another thing, Mahalo is "human edited" search results for the top queries, which is not a bad idea of course, but it is not intended to be a full search engine. Mahalo have indicated an interest in replacing their google search backup with our open source alternative, if we get to be good enough, which is obviously a far from foregone conclusion.

Wikia, the place to go for furry fan fiction

Wikia has been something of a dud. What Wikia really does is monetize fancruft. Their big wikis are for Star [Trek|Wars|Gate|Craft], Everquest, Marvel comics, Yu-Gi-Oh, and similar subjects. They're the resting place for fan articles thrown out of Wikipedia. [wikia.com]

Wikia's search engine, based on the user demographic they have now, is going to have great coverage of furry fan fiction. [wikia.com]

There's already a good manually-updated search engine. It's called Open Directory [dmoz.org]. It's quite useful as a data source for answering the question "what is this web site about"? It tends to run months behind changes to the web, since it's manually updated. While not many people query DMOZ manually, it's used by Yahoo, Google, etc. to get some basic information about a web site.

As an example of how great Wikia search is going to be, Wales suggested searching for "Tampa hotels". [techcrunch.com] The major search engines return too many bottom-feeder reseller and directory sites for searches like that. As I point out occasionally, we've already solved that problem over at SiteTruth [sitetruth.com], which looks for business legitimacy. Type in "Tampa hotels" there and watch it push the marginal sites to the bottom of the search results. We have that one handled.

Wikipedia works because people are willing to do substantial work for free for a non-profit organization. That doesn't work for a commercial business. You can get people to write about themselves (Myspace, Facebook, etc.) but beyond that, "crowdsourcing" doesn't go very far.

Funny thing is

that if this succeeds, and can race past MSN in terms of popularity, it will show to the world, that MS's gripe about Google truly was worthless. Of course, MS will use that to tell our congressmen that the glass is half empty, not half full.

The problem here is...

Wikipedia receives most of its traffic from its articles appearing in Google's search results, Wikipedia being relevant content, and Google being the top search engine.

How is Wikipedia to draw traffic to their search engine? Obviously not via Google, as search engines are content free on their own. Integrating it with Wikipedia? But again, Wikipedia is the end target, not a start point, so how could this work.

I don't think Wikipedia has the strategy or money for this to reach critical mass and show its potential, but it'll be interesting as an experiment.

It's called relevance feedback

and it's pretty useful. I don't know why everyone is complaining in the comments so far. This user-participates-in-ranking magic is not exactly news, and anyone who has studied or worked in information retrieval knows this. With a large enough number of benign participants, it should work.

And since people are bringing up Google as competition: Google Search has an estimated retrieval accuracy* of around 10%. Not very hard to beat, except that the Internet is a rather large document set. Have you ever browsed to the 50th page of results on Google? Good. Don't.

The problem is that to give decent results an engine needs time, and people are just not prepared to wait. That's why general purpose search engines on the web try to give you the best answer on the top hit. Results deteriorate a little (next 10) then improve again (next 20) then go completely nuts as you proceed. This fits the business plan, and almost everyone is happy. Google may have superb query processing and a decent Index system, but retrieval can be made to improve a lot if, say, there is an option to wait a little and get something better. Maybe Wikia can do this. If the users who get the most "insightful" (ergo time consuming) results get their feedback weighed more heavily than the point-and-click folks, this project can be very interesting.

*accuracy is a complicated metric that involves efficiency (fraction of retrieved that is relevant) and recall (fraction of relevant that is retrieved).

Scary Implications

I foresee someone hacking this system to return goatse as the #1 result for every search made.

Wikipedia fails

Wikipedia is falling apart because no one agrees on anything. Any additions are being deleted because they don't contain PC(pol.correct) language.

Why would the search website work?

I doubt it works.

so....

A great new searchengine.... and no url given....
Promoting stuff isn't their primary business I guess.
Hope they don't do ads....

Google already does this?

Doesn't Google do this already? AIUI the PageRank system looks at how a web site is linked to and connected to others. If a site is good an relevant, it will be related to other sites through links, blog posts etc, i.e. the net community in general. Perhaps that's even better than a group of self-appointed editors, and of course it's not the only factor Google uses.

Because of Google's anti-spam techniques this method is very hard to game. Better still, the defences are automatic. On Wikipedia, a lot of vandalism goes unnoticed for a long time because it's all mostly down to human oversight, with only basic anti-vandalism bots.

Personal preference ranking...

If they let me remove results from my personal ranking profile.... maybe with an option to show me how many total hits and a way to retrieve those alternate results if I need them.... then I'll be using it.

What I want is to get rid of things outside my daily context. I work in web development so when I do a search for "CSS 3 column hack" I don't want to get results related to some sport team abbreviated CSS where they have a coach whose a hack but likes to use a 3 column lineup.... stretching a but but you get the idea. More simply put I'd like to filter out results for anything that also ranks high in a variety of categories that have nothing to do with my daily tasks (fishing, fashion, first graders, fanboys... ) and it's not enough to let me use a -fishing in my search...

Also can I please remove a website from the results listing... ie: there are often high ranking sites that I've already bookmarked and read daily... I don't need to see them in my 'search' results, if i need to search them I'll use their own search or use website:www.domain.com 'keywords' (given that I'm using google for this one).

One-Upping the Least Bad

Funny to read this today, after I spent a couple of hours yesterday searching Google for something that doesn't exist -- a Plucker [plkr.org] type app for the iPod Classic. What struck me was just how badly Google performed. Any search containing the word "iPod" seems to return pages upon pages of blog entries about the (long since released) iPhone. What one tends to find with a Google search are a lot of loud, content-light blog entries, popping with ads, with short dashed-off articles broken across several pages. "Relevance" in Google seems to have the most to do with activity -- posts per day per site, repeated introductory blurbs on every page, modestly-trafficed forums devoid of meaningful discussion. Google does a pretty decent job with common searches, reasonably well with obscure searches, but very badly with the rest -- the middle of the long tail.

Google rose to prominence by being the best of a pretty weak set of players. It's still only the least bad solution, and there are a lot of things it does poorly. In classic AltaVista, you could type a few words of a song in quotes and find the title and lyrics. Type a long quoted string into Google, and you're likely to come up with nothing.

If Wikia manages to best Google in any type of search I'll applaud it. Search choices beyond Google and Trying to Be Google would be most welcome.

We already have this?

If we're talking about publicly-ranked search results, the results may expose more than we're comfortable with.

Wikipedia content is either right or wrong. It's not meant to be subjective, hence it can be patrolled and corrected. Now they want to apply it to subjective content; I don't see that making sense, albeit at first glance. User A is a technocrat who loves Monty Python. Hardly an isolated case. Use B is a 15yr old who likes whatever he/she likes this week. There's no "patrolling" this, except to address systematic abuse.

The concept is fine for slashdot, or any "closed" system, where the users generally share a common set of expectations. At /., I find all +5 content to be generally insightful, interesting, funny, etc. At least it seems so to me. Either I'm new here, or we've all seen Life of Brian. Whether that's utopia or not is another question altogether.

Expand this out to the general internet user, and the result will, of course, reflect the general focus of human society. That will be interesting, to say the least, though I'll bet $5 that anything entertainment- and religion-based will always be at the top of the results. Is that what people want? Ipso facto perhaps, but sure as hell not I.

Let's keep in mind that (no offence to anyone specific) ~80% of Americans believe in God, less than 50% subscribe to Darwin, ~30% believe in "UFOs, witches and astrology" (if you can believe this poll [physorg.com] that is). Of course, smart people believe weird things [sciam.com] too.

Add to this, that 81% [dogeatdogfilms.com] of those who have seen two or more "Police Academy" movies believe that O.J. is innocent, and you have a recipe for disaster.

Make search usable and teach people to use it

I had used many search search engines before Google was around and then started using Google when they became the most usable. I'd be happy to use a better search engine. The best way to win people over is to teach them to do better searches. I have seen hundreds of people use a search engine with one and two word searches and get millions of Google results. Then they need to read through pages of results. If they don't read pages of results, they get Google's top rated propaganda and not necessarily what they were looking for. Google rankings aren't bad, if you are specific, but Google's search results lost depth a while ago and I'm not sure when it happened. I should be able to find anything with the right search string, but I can't. I can get 50 Million pages, but when I drill down, I can't get 10 that I know exist and Google has indexed. There may be to many adword conflicts, so they don't show any ads or results :-)

Compete with Google and Yahoo? When pigs fly!

It's easier to find things on Wikipedia with Google than it is with the Wikipedia search... Good luck, Jimmy Wales. You're going to need it.

Re:Search Engine based in Wiki?

Yes, the internet has a clear left wing bias. I'm no fan.

Re:the worst idea ever

If it's an "open source" search engine and anyone can go and read the source of how it operates, everyone will know the secret to rigging their pages...

...but there's a big difference between "knowing the secret" and actually being able to break it. A "the secret" to breaking RSA is factoring really big numbers, but you can't actually do that.

It sounds like the "secret" to breaking this new system, like Wikipedia, would be to overwhelm the community that is guarding the data. We know that Wikipedia is working fine (for the most part), but things get a bit more complicated with search. Wikipedia, at least, knows when every single edit occurs. But with a whitelist or "reputation" list of URLs, there's no notification when domains (or subdomains or such) change hands (I think?), and re-vetting too often is probably untenable. And you don't really want results based upon the URL's reliability of staying on the whitelist, people might want relevance based on the most-recent data right now, sometimes even if it might disappear under a nasty registration/subscription barrier in a week.

But we'll see whether Wales is onto something good here, I guess. :)

Re:Why does everything have to be communist

sheeple? your lisp is cute.