slashdot's robots.txt

Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

Journal WhiteDragon's Journal: slashdot's robots.txt 3

Journal by WhiteDragon on Monday September 26, 2005 @03:32PM

Why is slashdot's robots.txt file allowing msn's and yahoo's indexing bots to crawl comments, user pages, and user journals, but google is prohibited? The stranger thing is that google's ad sense bot is allowed to do so.

This discussion has been archived. No new comments can be posted.

slashdot's robots.txt

Load All Comments

Search 3 Comments Log In/Create an Account

Comments Filter:

Maybe at Google's request? (Score:1)

by DisownedSky ( 905171 ) * writes:

However, some comments and JEs [google.com] are indexed by Google.
- Re:Maybe at Google's request? (Score:2)
  
  by WhiteDragon ( 4556 ) writes:
  
  interesting. Are they current comments, or old ones? If they are old, they may still be in google's index.
  - Re:Maybe at Google's request? (Score:2)
    
    by WhiteDragon ( 4556 ) writes:
    
    ok, a little investigation determines that they are in fact old comments.

There may be more comments in this discussion. Without JavaScript enabled, you might want to turn on Classic Discussion System in your preferences instead.

Today is a good day for information-gathering. Read someone else's mail file.

Journal WhiteDragon's Journal: slashdot's robots.txt 3

slashdot's robots.txt More Login

slashdot's robots.txt

Maybe at Google's request? (Score:1)

Re:Maybe at Google's request? (Score:2)

Re:Maybe at Google's request? (Score:2)

Slashdot Top Deals

Slashdot