New Search Engines

An anonymous reader wrote in to say "It seems that there's a new company out there called Fast Search and Transfer which is competing with Inktomi. They have a demo online at www.alltheweb.com and their engine seems to be ultra-fast. News about this is available here. Try out the demo, it is awesome what these guys have done." It is fast, but so far I've not had as good luck searching as with other engines. And the speed is probably largely due to the sparse HTML. But its not bad.

Re:Wow

[Madoc]

``Size isn't everything; but if you can have both size and relevancy, then you are able to please all types of users''

It's fast, but I found that it's relevance (and coverage) leaves a lot to be desired. It really pales in comparison to something like www.google.com [google.com] or www.altavista.com [altavista.com]. The similarities in setup to the latter are interesting as well (is it going to be a theme that hardware companies showcase their talents using search engines?).

As soon as they get much closer to their "all the web" idea, I'll take another look. But for now my default engines will stay.

Holy crap!

[gavinhall]

Posted by FascDot Killed My Previous Use:

I clicked the button and I had results before the mouse had fully un-clicked.

That's damn fast!

How could it get so fast?

[gavinhall]

Posted by sirtwist:

There's gotta be a reason apart from a fast machine. I have this feeling that the database ain't so big, making the searches incredibly speedy.

Still, it's a great speedy thing to see.

Re:How could it get so fast?

[gavinhall]

Posted by sirtwist:

Dammit.

I just read the article [yahoo.com]. At 80 million documents, I don't think my assertion is quite as valid.

Wow!

Prone to spamdexing

[Kurt Gray]

Try searching for "shareware" and check out the
results -- seems they don't discourage keyword
spammers!

Re:ftpsearch.lycos.com?

[knuth]

It looks to be by the same guys who created Fast FTP Search... which went downhill when it was bought by Lycos.

Not so FAST...

[martian]

As someone else commented, these people wrote the search engine behind ftpsearch.lycos.com. Which is fast. There are a few more reasons, apart from sparse HTML.

This engine asks for no cookies

The output is not in a table, so you see the results as they arrive in your browser, without having to wait for the whole table (in lynx it makes no difference :-)

The load is not very high yet, probably.

But having seen FTPsearch in action for the last 4-5 years, and having seen it always return results quickly, it wouldn't surprise me if alltheweb stayed fast.

Fast = FTPSearch, MP3Search et al

[hta]

This technology isn't very new.
Parts of it at least were developed at the Norwegian Institute of Technology, and used in the search engines FTPSEARCH [slashdot.org] and MP3Search.
But it's really nice to see them get the publicity they deserve.

Harald

Google is still better.

[stevenj]

I don't care how fast the search engine is, if it still returns 10,000 useless links. When are the other search engines going to realize that intelligent relevancy ranking is the single most important feature of a search engine?

The only search engine I know that does a good job at this is Google [google.com]. It is so good at finding relevant sites, I don't care if the response time is occasionally a little slow.

(Google uses a nice algorithm [google.com] where they gauge the "importance" of a page by how many other sites link to it.)

Beautiful!

[jonbrewer]

Damn this engine is fast! And I love how the searching syntax is spelled out on the first page. This is really phenomenal. Certainly my new default search engine.

Wow

[pluteus_larva]

that alltheweb.com is *fast*. good coverage too.

Duplicates?

[Bob McKenzie]

Egosurfing, I found it quite refreshing to NOT see my old web pages that died 4 months ago, yet found a large number of hits to people I'm apparantly related to. (gasp) Still no sign of my current web page tho. Curious. More time perhaps.

But did anyone else find there's number of identical links? (I'm used to metacrawler, so maybe this is normal for search engines?)

Re:FreeBSD

[Jae]

I wonder if these means that Dell is going to start supporting FreeBSD on their machines as well.

FreeBSD

[Agnomen]

Interesting to notice, their HTTP headers proclaim the site is running FreeBSD. It is nice to see FreeBSD and Linux being used in large commercial applications.

CNET Spamdexing???

[breser]

I find this really hard to believe but CNET is doing spamdexing [cnet.com].

Re:ftpsearch.lycos.com?

[cce]

yeah, off the bat i recognized the logo from the old ftpsearch.ntnu.no, now ftpsearch.lycos.com. are they a part of lycos, is lycos partnered with them, or what?

Re:Google is still better.

[LucaL]

The only gripe I have with google is that it's not always up to date. It seems a couple of month behind current events.

On the other hand, it's so good it's uncanny. If you search for "Linux", the first links it returns are:

Linux.org, Redhat, Debian.org, Linuxhq.com, ...

Try it on Altavista :)

Hardware-Based Searching

[scottsevertson]

The specs on the search engine are available at http://web.fast.no/product/search/d et.asp?id=34 [web.fast.no].

The press release doesn't exactly scream it out, but the search engine is actually just a little bit of software stuck on top of some pretty neat custom hardware. They call their chip the FAST PMC (Pattern Matching Chip), and their server is just your average (well, sort of average) high end server, with a buttload of those chips stuck on PCI cards.

The specs on the PMCs are available at http://web.fast.no/product/PMC/det.asp ?id=52 [web.fast.no].

FAST claims 100 MB/sec throughput on each chip, and each card has its own RAM (from 8 MB to 2 GB). The chips actually run at 100 mHz each, and even have support for RegEx matching (slightly limited).

From the specs:
A typical configuration will contain 4 to 8 plug-in cards per search node, and 16 or 32 chips on each card.


Overall, I'm pretty impressed - putting search capabilities into hardware is a pretty good idea, especially since so much of a modern processor is geared toward things like Floating Point calculations, which doesn't help text searching at all.


Scott Severtson
Software Developer
Auragen Communications
[e-mail]

Re:Lightning

[oyvindmo]

If a search takes exactly 0.0050 seconds, it's because it's cached. The engine caches all searches for some time, and if someone does the same search again, the results are simply fetched from the cache.

Database Servers?

[DonkPunch]

"...have constructed an advanced search capability using a high-performance, low-cost software/hardware combination...."

That little quote made me wonder what they were running. Why didn't they just say? They didn't seem at all shy about mentioning the Dell 4300s. :)

If the web servers are running FreeBSD, I wonder if it's also powering the database servers.

Maybe I'm reading too much into their vagueness here. It seems to me that companies which have close ties to MS (like Dell) are a little reluctant to trumpet the virtues of other operating systems too loudly. As a result, I tend to think:

#ifdef PRESS_RELEASE

Re:FreeBSD

[DonkPunch]

"...have constructed an advanced search capability using a high-performance, low-cost software/hardware combination...."

That little quote made me wonder what they were running. Why didn't they just say? They didn't seem at all shy about mentioning the Dell 4300s. :)

If the web servers are running FreeBSD, I wonder if it's also powering the database.

Maybe I'm reading too much into their vagueness here, but I tend to think:

#ifdef PRESS_RELEASE
#define HIGH_PERFORMANCE_LOW_COST Free_OS
#endif

(Sorry about the double-post. This is the one I meant to send.)

cool

[13th seer]

I found unique info to plagarize for the paper I'm writing.

FAST search and transfer is what?

[Hagbart Celine]

FAST is a norvegian company that specialize in fast algorithms for search and also transfer of video and images. Since FAST is a research company they are interested in selling their technology, not productifying it themselves. All the web is a marketing site for FAST technology and is probably going to be closed down after an eventual sale (according to norvegian newspapers).

The company has a couple of strange fellows working for them, there is former archeologist, medical doctors and statisticans. Guess a combination of their speciality fields spurred a really strange, but fast, algorithm.

Hope that will answer som questions.

By the way, the URL to fast is www.fast.no [www.fast.no].

Narcissist's dream engine

[Tofuhead]

I just did a search for my real name and came up with a ton of accurate links, more than any other engine in the past. Hot damn, ain't that fancy? =)

What can I say? I've bookmarked this engine.

Nice, but I just want a search engine!

[dblock]

Nice, fast powerful, whatever... yet another Altavista with maybe one day more relevant results.

Still, for us, common mortal beings and small company sites (50'000 to half a million pages), maybe a decent accessible search engine like Alkaline [vestris.com] is more than enough! It's free!

cheers

dB.

Re:database in memory?

[dblock]

That's how Alkaline [vestris.com] works. But for now an index of 179734 distinct word forms and 21309 pages eats ~40 MB together with the engine and serveral fast search structures. It grows slowly, so for 40'000 pages it will not get over 50-60 MB.

Anyhow, that means your next purchase of the first terabyte of RAM will make the shop happy :)

cheers

dB.

Re:How could it get so fast?

[dblock]

I guess it's purely cache methods. No database queries any more, pure hash hits in RAM.

cheers

dB.