Searching the 'Deep Web'

abysmilliard writes "Salon is running a story on next-generation web crawling technologies, specifically Yahoo's new paid "Content Acquisition Program." The article alleges that current search services like Google manage to access less than 1% of the web, and that the new services will be able to trawl the "deep web," or the 90-odd percent of web databases, forms and content that we don't see. Will access to this new level of specific information change how we deal with companies, governments and private insitutions?"

With the 10% that is crawled

[Trigun]

being pretty much total crap, I'd really hate to see the other 90%!

Porn!!!!!!!

[Anonymous Coward]

Best way to do a full deep search then

Damn ...

[Anonymous Coward]

I remember browsing the WWW directory in '93 and being able to scroll through all the sites on my VAX session at university. Are you telling me I am one of the few people who actually ever reached the end of the internet?

Oh yeah, a whole new pair of dimes

[stienman]

Will access to this new level of specific information change how we deal with companies, governments and private insitutions?"

Yeah. It means I'll be able to use someone else's credit card for more of my transactions, since finding credit cards, SSNs and other...uh...'deep web' stuff will be so much more accessable.

-Adam

Deep Web?

[dingo]

Why do I get the feeling that you will get a lot more search results for Linda Lovelace when searching the "Deep Web"

Deep web?

[hookedup]

Doesnt crap sink? Not sure I want to know what the other 90-odd percent is. After tubgirl, goatse, etc.. what else could possibly be next..

deep web?

[rjelks]

Is it just me, or does this sound like we're gonna get more pr0n when we search?

-

Re:Damn ...

[radicalskeptic]

Was the end guy hard? [bash.org]

BOFFINS ENHANCE SEARCH BY IGNORING robots.txt

[Anonymous Coward]

More shocking news at 11am !

Useless statistic of the week

[Alomex]

The article alleges that current search services like Google manage to access less than 1% of the web.

There's a useless statistic if you ask me.

I just wrote a cgi script that, upon requesting the url "http://bogus.com/nnnnn" returns a page with the text "nnnnn" where nnnnn is any number up to 1000 digits long. So there, I just added 10^1000 pages to the "deep web" of which google indexes none! (gasp).

So there, Google now indexes less than 0.001% of the deep web.

only missing 90 TB?

[DeathBunnyRanger]

the internet is only 90terrabytes?

that is what salon says, and I think that is bull, given my favorite porn site offers 20gigs of raunchy action.

More search results

[earthforce_1]

So instead of 5,234,169 search results returned, we will see 45,961,384 results?

Yippee!!!!!

Insight on the "deep web"

[saddino]

99% of the "deep web" probably looks like this [kentlaw.edu]. Indexable? Sure. Necessary? No.

Re:Deep Web?

[Rorschach1]

Doesn't observe it? It probably relies on it - tells you where the good stuff is!

Re:Deep web?

[djhertz]

I had not seen tubgirl before.. Google found it right away. How... damaging to my eyes and soul.