Comment new standard for robots.txt files - allow/disallow (Score 1) 837
All implications and conspiracy theories aside...
The new standard for robots.txt files (in place for about 2+ years now, IIRC) allows you to disallow the lot and then specify exceptions.
Example:
http://www.tesco.com/robots.txt
Benefit? You don't end up giving people a handy shopping list of places you don't want them poking around.
So what strikes me most about this little discovery is that whoever wrote this robots.txt file is a complete doofus.
Oh, and Oceania is at war with Eastasia. Oceania has always been at war with Eastasia.
Tim
http://www.bloggerheads.com/
The new standard for robots.txt files (in place for about 2+ years now, IIRC) allows you to disallow the lot and then specify exceptions.
Example:
http://www.tesco.com/robots.txt
Benefit? You don't end up giving people a handy shopping list of places you don't want them poking around.
So what strikes me most about this little discovery is that whoever wrote this robots.txt file is a complete doofus.
Oh, and Oceania is at war with Eastasia. Oceania has always been at war with Eastasia.
Tim
http://www.bloggerheads.com/
