Catch up on stories from the past week (and beyond) at the Slashdot story archive

 


Forgot your password?
Close
typodupeerror
×
The Internet Software

WebCrawler Turns 10 Today 136

Posted by timothy from the young-at-art dept.
Brian Pinkerton writes "WebCrawler, one of the first search engines on the 'Net, turns 10 today. You can read a short history of WebCrawler. When I wrote WebCrawler, one could do a credible job of crawling, indexing, and searching the Web from a single desktop PC. Today, the reality is a little bit different."
This discussion has been archived. No new comments can be posted.

WebCrawler Turns 10 Today More

WebCrawler Turns 10 Today

Comments Filter:

  • Guess this celebration... (Score:5, Funny)

    by oberondarksoul ( 723118 ) on Tuesday April 20, 2004 @07:11PM (#8923332) Homepage
    ...won't have an accompanying Google Doodle? [google.com]

  • e-mailing results (Score:4, Interesting)

    by qewl ( 671495 ) on Tuesday April 20, 2004 @07:12PM (#8923339)
    Ah the nostalgia of receiving search results via e-mail :)

  • Birthdays (Score:4, Funny)

    by 7Ghent ( 115876 ) on Tuesday April 20, 2004 @07:12PM (#8923341) Homepage
    Happy birthday to Webcrawler AND Hitler! Hurray!

  • They used to be my google.... (Score:5, Interesting)

    by wo1verin3 ( 473094 ) on Tuesday April 20, 2004 @07:12PM (#8923343) Homepage
    I remember when webcrawler was the only search engine I touched...

    In 1996 it was nice and simple [archive.org]. Then as the time went on [archive.org] it got a bit too cluttered for my liking [archive.org]. Now looks like they're trying to googlize themselves with the current interface [webcrawler.com].

  • Whoa! (Score:5, Interesting)

    by outZider ( 165286 ) on Tuesday April 20, 2004 @07:13PM (#8923347) Homepage
    Holy crap!

    I remember WebCrawler, but lost touch with it in around 1996, when I started religiously using AltaVista. They sure have changed a bit. ... but do they have any relevance anymore? They're owned by InfoSpace. :P
    • They're owned by InfoSpace

      I don't think InfoSpace has pwned anyone.

    • Yeah, i am totally in your shoes. Somebody above posted some way-back links from 96, and it brought back some fond memories of my first searchings. I went to the page now, and couldn't believe it was actually webcrawler! I moved from webcrawler to hotbot to google. I actually thought webcrawler had just sort of dropped off the face of the net, but i guess that never really happens
    • I remember WebCrawler, but lost touch with it in around 1996, when I started religiously using AltaVista. They sure have changed a bit. ... but do they have any relevance anymore? They're owned by InfoSpace. :P

      A lot of usenet people that I knew dropped WebCrawler shortly after the AOL deal went through.

      AltaVista then ended up being the search engine that most people recommended.

  • Wow (Score:5, Interesting)

    by z0ink ( 572154 ) on Tuesday April 20, 2004 @07:13PM (#8923355)
    Does anybody else remember getting a WebCrawler promotional CD 10 years ago? I didn't even have a CD-ROM then!
    • I think we played frizbee with ours... seriously. But I do remember using Web Crawler quite frequently at the Internet Cafe in Kingston. Oh those were the days.... :-)

      I can remember really digging the simple search interface.

      • Re:Wow (Score:2)

        by BTWR ( 540147 )
        everyone always suggests this as a possible use of AOL cds, but I've NEVER been able to accurately throw one more than 3 feet before it turns from horizontal to vertical and crashes (and hurts!). Any tips?
        • > Any tips?

          Put your rage in it and then throw. Oh, and flick your wrist. Oh, and take the packaging off first! :-)

  • I remember using Webcrawler before google... (Score:5, Insightful)

    by John Seminal ( 698722 ) on Tuesday April 20, 2004 @07:17PM (#8923384) Journal
    It was a good search engine. I dunno why I stopped using it, I think it was a bit on the slow side and Google had more pages.

    Heck, while reminiscing, I remember when excite was my start page, and when I used them for email. I remember they were the first "start" page to have groups. I stopped using them 4 years ago when their email stopped working.

    I guess if anything, we can learn the web is not going to be the same in 5 years as it is today. My question is, "is it better"? Personally, I think it was better back in the day. I would like to see a search engine that does not display any spam or sales or sex sites as hits. I now do most of my searches on google doing "search parameters site:edu".

    • Re:I remember using Webcrawler before google... (Score:5, Interesting)

      by Anonymous Coward on Tuesday April 20, 2004 @07:33PM (#8923506)
      I think I remember why I left WebCrawler.

      WebCrawler was simple and effective. But then AltaVista emerged. I started using AltaVista.digital.com, and from there WebCrawler went down hill - lots of advertising and junk that kind of made me hate it. What was once seemless and simple became noisy.

      I used AltaVista for a number of years, but once again advertising got the best of it. It turned super-sophisticated, with a lot of advertising fluff and "features". Altavista was becoming overly commercialized. They had a "simple" version that was better (I forget the name [begins with an "R"?]), but soon the result sets were scewed towards advertisers and abusers.

      In 2001, I made the switch to Google. It was everything that WebCrawler once was in terms of ease of use and quality of results. I've been more or less happy with Google ever since.

  • /me 's jaw hits the floor (Score:3, Funny)

    by Stalin ( 13415 ) on Tuesday April 20, 2004 @07:18PM (#8923388)
    I can't believe it is even still around.

  • Then and now... (Score:5, Funny)

    by jdreed1024 ( 443938 ) on Tuesday April 20, 2004 @07:19PM (#8923403)
    When I wrote WebCrawler, one could do a credible job of crawling, indexing, and searching the Web from a single desktop PC. Today, the reality is a little bit different.

    No kidding. Back then, one could serve a website from most any machine, and it would be there for all to see. Today only the largest websites can avoid a slashdotting with only 9 posts in the thread.

  • Holy search engines batman! (Score:3, Interesting)

    by ylikone ( 589264 ) on Tuesday April 20, 2004 @07:20PM (#8923405) Homepage
    Who uses webcrawler anymore? I didn't even know they still exist. Anybody remember opentext.com search?

  • Birthday party (Score:5, Funny)

    by jacobhoupt ( 728382 ) on Tuesday April 20, 2004 @07:20PM (#8923409)
    I'll be hosting my tenth annual WebCrawler birthday party tonight in the back of my Yugo.

    Feel free to drop in, there should be plenty of seating available for those interested.

  • my new hero (Score:5, Funny)

    by theMerovingian ( 722983 ) on Tuesday April 20, 2004 @07:20PM (#8923411) Journal

    Some guys are too cool for their own good. Brian Pinkerton has the domain 'thinkpink.com', AND he wrote his own search engine.

    I bet he even has a 3-digit UID, a beowulf cluster of Xboxes running linux, and he sold all his stock options during the bubble. :)

  • Already Slashdotted (Score:3, Funny)

    by MrRuslan ( 767128 ) on Tuesday April 20, 2004 @07:22PM (#8923429)
    Here is the google cache
    http://216.239.39.104/search?q=cache:-vPR77Hq9OYJ: www.thinkpink.com/bp/WebCrawler/History.html+&hl=e n&ie=UTF-8
  • ah yes, i remember the old days, search engines like webcrawler, altavista, magellan, and infoseek. in those days wouldn't help you find what you want most of the time. now with google we need not worry :)

  • When did they give up.... (Score:5, Interesting)

    by David Hume ( 200499 ) on Tuesday April 20, 2004 @07:23PM (#8923435) Homepage

    ...on their own web search technology and become a metasearch engine? From the WebCrawler About Page [webcrawler.com]:

    WebCrawler uses innovative metasearch technology to search the Internet's top search engines, including Google, Yahoo, Ask Jeeves, About, Teoma, FindWhat, LookSmart, and many more.


    With one single click, WebCrawler searches the best results from the combined pool of the world's leading search engines -- instead of results from only one single search engine.

    And WebCrawler makes it easy to refine your search so you can find the most meaningful results right away. No wonder it's a leader in the search industry.


    Was it 2001? The History [thinkpink.com] states:

    2001 InfoSpace acquires WebCrawler. Excite, now Excite@Home, went belly up. In the bankruptcy, Infospace acquired WebCrawler. Today Infospace runs WebCrawler as a meta-search engine. And they've given Spidey a new name and turned him purple!


    Oh, and if it is not being otherwise used, has the code for the WebCrawler spider been open-sourced? :)

    • There's MetaCrawler [metacrawler.com]. If my memory serves me correctly, it appeared before WebCrawler went to this format.

      I honestly don't remember the first time I saw MetaCrawler (but it used to be much simpler back then!) so I don't know if it predates Google. WebCrawler's idea however is not new, AFAIK.

      • Metacrawler easily predates Google.
        I was using it in, like, '95 or '96. (Webcrawler *was* my first, though)

        I seem to remember it didn't even have a domain name back then, it was a page on some university site.

        I know Google has some history, but I only started using Google Beta sometime in '99 or so.
        I'm sure the initial engine wasn't around in '95.

    • The more things change... (Score:4, Interesting)

      by Old Man Kensey ( 5209 ) on Tuesday April 20, 2004 @08:51PM (#8924137) Homepage
      Originally there was WebCrawler (among others). In late 1996, AOL acquired WebCrawler and turned it into AOL Netfind. Later, apparently, Excite bought it from AOL, made it a separate service, and Excite became the engine that powered AOL Netfind. After that apparently InfoSpace bought it in the Excite sell-off.

      But after AOL bought it I lost track of it, because it started sucking (returning lots more stale links than before), and altavista.digital.com burst upon the scene (anyone else remember "kayak sailing San Juan islands"?).

      My guess would be that the meta-search switch initially happened when Excite bought them.

  • Boy, does this take me back... (Score:5, Interesting)

    by Faust7 ( 314817 ) on Tuesday April 20, 2004 @07:23PM (#8923440) Homepage
    ...to the days when the search engine market resembled the microcomputer market of the '80s. Several competitors, all with (roughly) the same market share, each with a certain number of hits that the others didn't have. I had to use at least a few of them to assure myself that I was getting something reasonably close to what the whole Web could offer on my search topic (even though no search engine comes close to penetrating all of the pages out there).

    If I was looking for something, I'd query Lycos, AltaVistas, Infoseek, Excite, Webcrawler, and Magellan. And, later on, Google. Vastly different results, site designs, site objectives. I won't say it was the most streamlined, elegant experience, but it was kind of fun.

  • Wow. Just. Wow. (Score:5, Interesting)

    by TWX ( 665546 ) on Tuesday April 20, 2004 @07:26PM (#8923460)
    I remember using Webcrawler back when I got my first 14.4 Slirp connection back in 1994. It was the only way to search!

    and then came the marvels of altavista.digital.com.

    I'm so glad that google came along...

  • WebCrawler was the best, back in the day... (Score:1, Interesting)

    by Anonymous Coward
    WebCrawler was the best search engine, back in the day when search engines were all new and Yahoo wasn't a search engine but a human-moderated list of sites.

    --
    Callas

  • Wow - the 1996 wayback WebCrawler page STILL WORKS (Score:5, Interesting)

    by Anonymous Coward on Tuesday April 20, 2004 @07:29PM (#8923482)
    http://web.archive.org/web/19961023234707/http://w ww.webcrawler.com/

    Presumably connects to the current crawler which still accepts the old format :)

    --
    Callas

  • First query? (Score:5, Funny)

    by Xzzy ( 111297 ) <sether@@@tru7h...org> on Tuesday April 20, 2004 @07:34PM (#8923516) Homepage
    So who remembers the first search query they typed into Webcrawler?

    I was just crawling out of the gopher world, a short period where I was getting turned on to the web but there was no way to find links, almost everything came through the university homepage or word of mouth. Then someone pointed me to webcrawler.

    What did I search for first? "fart jokes". No kidding.

    "boobs" was second.
  • Skynet!

    Engineer: ARGGHHH... IT'S ALIIIIIVE... <BANG> <CRACK>

    <STATIC>

  • WebCrawler on NeXTStep - before Open Source (Score:5, Interesting)

    by ben_kelley ( 234423 ) on Tuesday April 20, 2004 @07:39PM (#8923543)
    It is scary to think that at one point I e-mailed the WebCrawler people to ask them how it worked. In response they sent me a copy of the source (Objective C for NeXT) so I could compile it up on my NeXT PC (I had a "black" NeXT - 68000 based) to index my intranet web server.

    I doubt that someone like Google would send you a copy of their source these days - even if you asked nicely.

    I could never get it to compile, and I deleted it long ago, but I kind of wish I had kept it now. An interesting piece of internet history.

  • DNS failure (power outage) (Score:1, Insightful)

    by Anonymous Coward
    Try http://209.24.201.206/bp/WebCrawler/History.html for the history.

  • Worth Remember (Score:2, Funny)

    by Anonymous Coward
    I've reminisced before on slashdot about the beautiful geeky girl that introduced me to hotbot. [hotbot.com] Glasses, long blond hair, full breasts... cute sandals, short skirts... those silk panties.

    Fuck WebCrawler. hotbot. [hotbot.com]

  • The WebCrawler Search Voyeur (Score:5, Funny)

    by Faust7 ( 314817 ) on Tuesday April 20, 2004 @07:46PM (#8923584) Homepage
    Anyone remember the WebCrawler Search Voyeur?

    It was a little Java applet that sat on your screen and displayed the pseudo-real-time search queries of other people.

    When I was a computer lab monitor at my college, we used to note in the log book any particularly amusing queries that we'd seen.

    "hairy woman"... "squirrel torture"... "tom AND cruise AND foot AND odor"... "asian girl underage spanking"...
    • I'll never forget reading this entry in the monitor log book:

      "Search Voyeur query of the day:
      'Why does poop stink?'"

    • Re:The WebCrawler Search Voyeur (Score:4, Funny)

      by intangible ( 252848 ) on Tuesday April 20, 2004 @08:46PM (#8924100) Homepage
      I remember that it wouldn't show every search of course, but you could verify it was working by searching for the same phrase over and over again. About 10 seconds later, you could see your search phrase. You could actually use it to communicate with other people, albeit a little slow, but it was amusing. I would type in silly things just so others watching the voyeur would see them.
      I bet you guys recorded some of my stuff :P

    • Re:The WebCrawler Search Voyeur (Score:3, Informative)

      by Anonymous Coward
      It's still there in a slightly different incarnation.... http://www.metaspy.com [metaspy.com]

    • Re:The WebCrawler Search Voyeur (Score:1, Interesting)

      by Anonymous Coward
      yeah, I remember that. I think I ended up staring at the screen for the afternoon with my jaw on the floor in between giggle fits...

      It was like a really horrible glimpse inside the mind of my fellow man... but, funny...
  • I just did a search on webcrawler for "digital camera" and the results where 70% pay-per-click advertising with a small and hardly noticable "Sponsored by:" disclaimer. Worse yet the paid links are intermigled with the indexed hits.

    Looks like Webcrawler is now more of a pay-per-click dispensor than a search engine... No thanks!

    I think google has done a good job of clearly identified what is relevant and what is paid for.

  • public search engine (Score:5, Interesting)

    by jacquesm ( 154384 ) <j@NoSpam.ww.com> on Tuesday April 20, 2004 @07:57PM (#8923645) Homepage
    I'd happily contribute cash to a publicly funded and publicly run search engine.

    Anyone game ?

  • I remember the exact day (Score:2, Funny)

    by Anonymous Coward
    The exact day that I stopped using webcrawler. It happened to coincide with the day that AOL was announced as the new owner.

  • I didn't even know they were around anymore. (Score:3, Interesting)

    by Captain Rotundo ( 165816 ) on Tuesday April 20, 2004 @08:01PM (#8923686) Homepage
    And the odd part is I don't even remember the interface being as cluttered as the very early one linked through the archive in an earlier post. I suppose I moved on very early, although I remember when as far as I was concerned they were the only game in town.

  • Takes me back (Score:3, Interesting)

    by SuperBigGulp ( 177180 ) on Tuesday April 20, 2004 @08:11PM (#8923812)

    I remember using WebCrawler on my very first SLIP dial up account and thinking "How cool is this?" I had used AOL for a couple years prior but was hoping trade in their UI (and limitations) for Netscape. The funny thing is that I wasn't sure if I could find enough content on the web.

    Also a great testament to the original design and concept that search engines still look and work a lot like WebCrawler, 10 years on.

    Happy birthday, and thanks for the walk down 32K memory lane

  • I seem to remember that the WebCrawler site used to have a "Powered By NEXTSTEP" badge on it. I can't verify this with web.archive.org as it doesn't seem to go back that far (I started using WebCrawler in 1995). I can't RTFA at the moment, does anyone know what sort of hardware powered the WebCrawler site originally? Did it run on black NeXT hardware? White box PCs running NeXTSTEP? Did they ever utilize the WebObjects framework that NeXT (and later, Apple) used to sell?
    • Sounds like white boxes instead...

      "In the current implementation, the WebCrawler builds an index at the rate of about 1000 documents an hour on a 486-based PC running NEXTSTEP."

      "The full-text index is currently based on NEXTSTEP's IndexingKit [NeXT]"

      - from Experiences with Webcrawler [uiuc.edu]

      I think Webcrawler used CERN's WWW library, but I can't say this made it's way into WebObjects.

  • Hardly one of the first (Score:5, Informative)

    by btempleton ( 149110 ) on Tuesday April 20, 2004 @09:22PM (#8924343) Homepage
    Internet searching way predates 1994. Archie by Peter Deutsch (the one from Montreal, not the American one) was one of the most popular applications on the internet in the 80s. The http search engines like Webcrawler and Lycos came much, much later on internet time scales.
  • I seem to remember that before WebCrawler there was actually a "big" search engine run by a non-profit. For the life of me I can't remember what it was, but I seem to remember one day going "Wow, this webcrawler thing is great, I'm never touching [whatever] again."

    Of course a few years later I said "Wow, this AltaVista thing is great. I'm never touching WebCrawler again." And then I went "Wow, this Google thing is great. I'm never touching AltaVista again."

  • Wow... (Score:4, Interesting)

    by }InFuZeD{ ( 52430 ) on Tuesday April 20, 2004 @10:21PM (#8924825) Homepage
    I think WebCrawler was my first search engine ever...

    From there I graduated to MetaCrawler, which parsed WebCrawler and all the other currently popular web search engines at the time.

    For some reason or another MetaCrawler started sucking and I used InfoSeek for quite some time... then they were acquired by Go.com and it went downhill from there.

    I remember what I'd search the internet for back in those days tho. It was always "jedi knight" and "giga pets" (remember those cute tamagotchi rip-offs? =p)

    • More WebCrawler History (Score:2, Interesting)

      by Anonymous Coward
      I used to be one of the Excite@Home engineers who looked after Webcrawler. WebCrawler and the Excite front end all belonged to the same code base called My Excite Start Page (known internally as MESP at Excite).

      The WebCrawler at Excite was pretty much an unsupported product when I was there. All I ever did were maintenance releases, never any new stuff for WebCrawler. WebCrawler was actually the Excite front end, except it had the WebCrawler logos instead of Excite.

      The search engine was the Excite search

  • WebCrawler Sale Sensation (Score:4, Interesting)

    by Anonymous Coward on Tuesday April 20, 2004 @11:44PM (#8925315)
    WebCraweler's Brian Pinkerton formerly worked at NeXT, and I remember being in the the NeXT kitchen when news arrived in 1995 of his sale of WebCrawler to AOL. The sale price was around $1 million, and everyone was absolutely awed that a software company could sell for so much. This marked the beginning for me of the dot-com era: Just a few month later, other companies started or run by ex-NeXTers sold for millions, then tens of millions, and at least one for hundreds of million. Soon after that, NeXT CEO Jobs took Pixar through an IPO, for a personal gain of about $1 billion!

  • Ah... back in the day (Score:3, Interesting)

    by StefanSavage ( 454543 ) on Wednesday April 21, 2004 @01:34AM (#8925940)
    I remember back in 1994 WebCrawler was running on three machines in the corner of Sieg Hall 433. They were rigged up so one could reboot the others via a serial line, but occassionally that machine would crash too. That was when Brian would call in and say "Hey, Webcrawler is hung. Could you go reboot it?". I'm guessing this doesn't happen much at Google...

  • nope (Score:5, Informative)

    by millette ( 56354 ) <robin@@@millette...info> on Wednesday April 21, 2004 @03:01AM (#8926307) Homepage Journal
    You just need 8 desktop machines [gigablast.com] and you can index a 10th of what google does. From a recent article:
    Gigablast runs on eight desktop machines, each with four 160-GB IDE hard drives, two gigs of RAM, and one 2.6-GHz Intel processor. It can hold up to 320 million Web pages (on 5 TB), handle about 40 queries per second and spider about eight million pages per day. Currently it serves half a million queries per day to various clients, including some meta search engines and some pay-per-click engines.
    I also read it was going to expand it's index this year, but I wasn't able to find where I read that.

Slashdot Top Deals

"Everything should be made as simple as possible, but not simpler." -- Albert Einstein

Close