Comment Re:what! no Google? (Score 1) 285
"... we can find what we are looking for, whether those with the domains like it or not... "
Actually, those with the domains can prevent Google from indexing certain pages or even the entire site.
Search engines like Google use a technology called Robots. Search engine robots automatically scour the web returning information about each page they visit to the search engine's indexing search. This is then cache and correlated in a database.
Robots can be blocked by using a robots.txt file in the root directory of the web site. So, if your domain is foo.com, the corresponding robots.txt file would be at http://foo.com/robots.txt
Check out Web Server Administrator's Guide to the Robots Exclusion Protocol
Actually, those with the domains can prevent Google from indexing certain pages or even the entire site.
Search engines like Google use a technology called Robots. Search engine robots automatically scour the web returning information about each page they visit to the search engine's indexing search. This is then cache and correlated in a database.
Robots can be blocked by using a robots.txt file in the root directory of the web site. So, if your domain is foo.com, the corresponding robots.txt file would be at http://foo.com/robots.txt
Check out Web Server Administrator's Guide to the Robots Exclusion Protocol
