WebCrawler Turns 10 Today 136
Brian Pinkerton writes "WebCrawler, one of the first search engines on the 'Net, turns 10 today. You can read a short history of WebCrawler. When I wrote WebCrawler, one could do a credible job of crawling, indexing, and searching the Web from a single desktop PC. Today, the reality is a little bit different."
Guess this celebration... (Score:5, Funny)
e-mailing results (Score:4, Interesting)
Re:e-mailing results (Score:5, Informative)
Simply email google@capeclear.com with the search terms in the subject line, you will soon recieve a response with the results. I think there is a limit to how many times a day you can use this, but I cannot find the link to the project webpage.
And from there it goes to spam lists, right? (Score:2)
Re:And from there it goes to spam lists, right? (Score:2)
So you don't post to mailing lists either?
BTW, the google-by-mail thing's webpage is here [capeclear.com].
Birthdays (Score:4, Funny)
Re:Birthdays (Score:4, Funny)
Re:Birthdays (Score:1, Funny)
Barney Gumble too! (Score:1)
Godwin's Law Invoked (n/t) (Score:1)
Re:Birthdays (Score:2)
They used to be my google.... (Score:5, Interesting)
In 1996 it was nice and simple [archive.org]. Then as the time went on [archive.org] it got a bit too cluttered for my liking [archive.org]. Now looks like they're trying to googlize themselves with the current interface [webcrawler.com].
Re:They used to be my google.... (Score:5, Funny)
Very cool research tool.
Re:They used to be my google.... (Score:2, Funny)
Re:They used to be my google.... (Score:2)
> >
> > Very cool research tool.
>
> Pantyhose research? One moderation point left and there's no option for +/-1 WTF!?
For every kink, fetish, and perversion, there exists at least one adherent with a website. Proof is left as an exercise for the Internet.
And thus, web crawlers were born.
Re:They used to be my google.... (Score:1)
Re:They used to be my google.... (Score:2)
Gee. Pimple out folks.
Re:They used to be my google.... (Score:1)
I really like the... (Score:2)
Those things may have urban legends surrounding them or whatever, but they are GODDAMN SCARY!!!
Re:They used to be my google.... (Score:2)
Touched what? No, never mind. I dont want to know.
Re:They used to be my google.... (Score:1)
One sort of interesting note:
sometime between May 6, 1999 [archive.org] and Oct. 9, 1999 [archive.org], WebCrawler stopped pitching the Netscape Now! and Microsoft Internet Explorer buttons at the bottom of the page.
an interesting milestone, to say the least
Whoa! (Score:5, Interesting)
I remember WebCrawler, but lost touch with it in around 1996, when I started religiously using AltaVista. They sure have changed a bit.
Re:Whoa! (Score:1)
I don't think InfoSpace has pwned anyone.
Re:Whoa! (Score:1)
Re:Whoa! (Score:1)
A lot of usenet people that I knew dropped WebCrawler shortly after the AOL deal went through.
AltaVista then ended up being the search engine that most people recommended.
Wow (Score:5, Interesting)
Re:Wow (Score:1)
I can remember really digging the simple search interface.
Re:Wow (Score:2)
Rage (Score:1)
Put your rage in it and then throw. Oh, and flick your wrist. Oh, and take the packaging off first!
I remember using Webcrawler before google... (Score:5, Insightful)
Heck, while reminiscing, I remember when excite was my start page, and when I used them for email. I remember they were the first "start" page to have groups. I stopped using them 4 years ago when their email stopped working.
I guess if anything, we can learn the web is not going to be the same in 5 years as it is today. My question is, "is it better"? Personally, I think it was better back in the day. I would like to see a search engine that does not display any spam or sales or sex sites as hits. I now do most of my searches on google doing "search parameters site:edu".
Re:I remember using Webcrawler before google... (Score:5, Interesting)
WebCrawler was simple and effective. But then AltaVista emerged. I started using AltaVista.digital.com, and from there WebCrawler went down hill - lots of advertising and junk that kind of made me hate it. What was once seemless and simple became noisy.
I used AltaVista for a number of years, but once again advertising got the best of it. It turned super-sophisticated, with a lot of advertising fluff and "features". Altavista was becoming overly commercialized. They had a "simple" version that was better (I forget the name [begins with an "R"?]), but soon the result sets were scewed towards advertisers and abusers.
In 2001, I made the switch to Google. It was everything that WebCrawler once was in terms of ease of use and quality of results. I've been more or less happy with Google ever since.
Re:I remember using Webcrawler before google... (Score:5, Informative)
Re:I remember using Webcrawler before google... (Score:2)
/me 's jaw hits the floor (Score:3, Funny)
Then and now... (Score:5, Funny)
No kidding. Back then, one could serve a website from most any machine, and it would be there for all to see. Today only the largest websites can avoid a slashdotting with only 9 posts in the thread.
Re:Then and now... (Score:5, Funny)
Re:Then and now... (Score:2)
Re:Then and now... (Score:1, Insightful)
Re:Then and now... (Score:1)
Re:Then and now... (Score:3, Informative)
Not true, see Surviving Slashdotting with a Small Server [slashdot.org]. Lots of people tried to bring it down (see comments), but it survived with no trouble at all.
Holy search engines batman! (Score:3, Interesting)
Birthday party (Score:5, Funny)
Feel free to drop in, there should be plenty of seating available for those interested.
my new hero (Score:5, Funny)
Some guys are too cool for their own good. Brian Pinkerton has the domain 'thinkpink.com', AND he wrote his own search engine.
I bet he even has a 3-digit UID, a beowulf cluster of Xboxes running linux, and he sold all his stock options during the bubble.
Re:my new hero (Score:2)
Re:OT, your sig (Score:3, Funny)
Re:OT, your sig (Score:1)
Already Slashdotted (Score:3, Funny)
http://216.239.39.104/search?q=cache:-vPR77Hq9OYJ
Well isn't that ironic (Score:5, Funny)
Go figure.
Re:Well isn't that ironic (Score:3, Insightful)
Re:Already Slashdotted (Score:1)
E.
old search engines (Score:2)
When did they give up.... (Score:5, Interesting)
Was it 2001? The History [thinkpink.com] states:
Oh, and if it is not being otherwise used, has the code for the WebCrawler spider been open-sourced?
Re:When did they give up.... (Score:3, Informative)
I honestly don't remember the first time I saw MetaCrawler (but it used to be much simpler back then!) so I don't know if it predates Google. WebCrawler's idea however is not new, AFAIK.
Re:When did they give up.... (Score:1)
I was using it in, like, '95 or '96. (Webcrawler *was* my first, though)
I seem to remember it didn't even have a domain name back then, it was a page on some university site.
I know Google has some history, but I only started using Google Beta sometime in '99 or so.
I'm sure the initial engine wasn't around in '95.
The more things change... (Score:4, Interesting)
But after AOL bought it I lost track of it, because it started sucking (returning lots more stale links than before), and altavista.digital.com burst upon the scene (anyone else remember "kayak sailing San Juan islands"?).
My guess would be that the meta-search switch initially happened when Excite bought them.
Boy, does this take me back... (Score:5, Interesting)
If I was looking for something, I'd query Lycos, AltaVistas, Infoseek, Excite, Webcrawler, and Magellan. And, later on, Google. Vastly different results, site designs, site objectives. I won't say it was the most streamlined, elegant experience, but it was kind of fun.
Oh, Yahoo too. (Score:3, Interesting)
Interestingly, their look has changed very, very little from their olden days.
Re:Oh, Yahoo too. (Score:3, Interesting)
Re:Speaking of Yahoo (Score:2)
Re:Imagine if Copernic had become the standard (Score:2)
Re:Boy, does this take me back... (Score:2)
That's why metasearch engines popped up. Can't remember any of their names though... Metacrawler maybe? I can't bring myself to Google for them.
Re:Boy, does this take me back... (Score:2)
Re:Boy, does this take me back... (Score:2)
Wow. Just. Wow. (Score:5, Interesting)
and then came the marvels of altavista.digital.com.
I'm so glad that google came along...
WebCrawler was the best, back in the day... (Score:1, Interesting)
--
Callas
Wow - the 1996 wayback WebCrawler page STILL WORKS (Score:5, Interesting)
Presumably connects to the current crawler which still accepts the old format
--
Callas
Re:Wow - the 1996 wayback WebCrawler page STILL WO (Score:2, Interesting)
I have NO idea how that space got in there...
--
Callas
Re:Wow - the 1996 wayback WebCrawler page STILL WO (Score:2, Informative)
Not your fault. Slashcode does that itself whenever there's a long enough unbroken string of characters, to stop page-widening posts.
Re:Wow - the 1996 wayback WebCrawler page STILL WO (Score:1)
Presumably connects to the current crawler which still accepts the old format :)
Ya, but I followed the link to get my free copy of Internet Explorer that is advertised on the page and got
Whatever that means.
First query? (Score:5, Funny)
I was just crawling out of the gopher world, a short period where I was getting turned on to the web but there was no way to find links, almost everything came through the university homepage or word of mouth. Then someone pointed me to webcrawler.
What did I search for first? "fart jokes". No kidding.
"boobs" was second.
Google's next step... (Score:2, Funny)
Engineer: ARGGHHH... IT'S ALIIIIIVE... <BANG> <CRACK>
<STATIC>
WebCrawler on NeXTStep - before Open Source (Score:5, Interesting)
I doubt that someone like Google would send you a copy of their source these days - even if you asked nicely.
I could never get it to compile, and I deleted it long ago, but I kind of wish I had kept it now. An interesting piece of internet history.
Re:WebCrawler on NeXTStep - before Open Source (Score:4, Funny)
Re:WebCrawler on NeXTStep - before Open Source (Score:1, Funny)
Re:WebCrawler on NeXTStep - before Open Source (Score:2, Insightful)
The next best thing.
search appliance [google.com]
Re:WebCrawler on NeXTStep - before Open Source (Score:5, Interesting)
the first web server
Webcrawler
Doom and DoomII
Pretty good for a machine that only sold ~70,000 units total, not including the versions of NEXTSTEP for ix86/SPARC/PA-RISC.
I still have a Color NeXTStation stashed away in a closet. I was using it as a print server till about two years ago.
DNS failure (power outage) (Score:1, Insightful)
Worth Remember (Score:2, Funny)
Fuck WebCrawler. hotbot. [hotbot.com]
The WebCrawler Search Voyeur (Score:5, Funny)
It was a little Java applet that sat on your screen and displayed the pseudo-real-time search queries of other people.
When I was a computer lab monitor at my college, we used to note in the log book any particularly amusing queries that we'd seen.
"hairy woman"... "squirrel torture"... "tom AND cruise AND foot AND odor"... "asian girl underage spanking"...
Almost forgot this (Score:2)
"Search Voyeur query of the day:
'Why does poop stink?'"
Re:The WebCrawler Search Voyeur (Score:4, Funny)
I bet you guys recorded some of my stuff
Re:The WebCrawler Search Voyeur (Score:3, Informative)
Re:The WebCrawler Search Voyeur (Score:1, Interesting)
It was like a really horrible glimpse inside the mind of my fellow man... but, funny...
Webcrawler is a crawler no more.... (Score:2, Redundant)
Looks like Webcrawler is now more of a pay-per-click dispensor than a search engine... No thanks!
I think google has done a good job of clearly identified what is relevant and what is paid for.
public search engine (Score:5, Interesting)
Anyone game ?
Re:public search engine (Score:1, Informative)
Re:public search engine (Score:3, Funny)
I remember the exact day (Score:2, Funny)
I didn't even know they were around anymore. (Score:3, Interesting)
Takes me back (Score:3, Interesting)
I remember using WebCrawler on my very first SLIP dial up account and thinking "How cool is this?" I had used AOL for a couple years prior but was hoping trade in their UI (and limitations) for Netscape. The funny thing is that I wasn't sure if I could find enough content on the web.
Also a great testament to the original design and concept that search engines still look and work a lot like WebCrawler, 10 years on.
Happy birthday, and thanks for the walk down 32K memory lane
Wasn't WebCrawler "Powered By NEXTSTEP" ? (Score:2)
Re:Wasn't WebCrawler "Powered By NEXTSTEP" ? (Score:2)
"In the current implementation, the WebCrawler builds an index at the rate of about 1000 documents an hour on a 486-based PC running NEXTSTEP."
"The full-text index is currently based on NEXTSTEP's IndexingKit [NeXT]"
- from Experiences with Webcrawler [uiuc.edu]
I think Webcrawler used CERN's WWW library, but I can't say this made it's way into WebObjects.
Hardly one of the first (Score:5, Informative)
The one before WebCrawler? (Score:2, Insightful)
Of course a few years later I said "Wow, this AltaVista thing is great. I'm never touching WebCrawler again." And then I went "Wow, this Google thing is great. I'm never touching AltaVista again."
Wow... (Score:4, Interesting)
From there I graduated to MetaCrawler, which parsed WebCrawler and all the other currently popular web search engines at the time.
For some reason or another MetaCrawler started sucking and I used InfoSeek for quite some time... then they were acquired by Go.com and it went downhill from there.
I remember what I'd search the internet for back in those days tho. It was always "jedi knight" and "giga pets" (remember those cute tamagotchi rip-offs? =p)
More WebCrawler History (Score:2, Interesting)
The WebCrawler at Excite was pretty much an unsupported product when I was there. All I ever did were maintenance releases, never any new stuff for WebCrawler. WebCrawler was actually the Excite front end, except it had the WebCrawler logos instead of Excite.
The search engine was the Excite search
WebCrawler Sale Sensation (Score:4, Interesting)
Ah... back in the day (Score:3, Interesting)
nope (Score:5, Informative)
Re:world wide worm? (Score:2, Informative)