Google Blogoscoped

Forum

MSN robot.txt cached by google

/pd [PersonRank 10]

Saturday, August 12, 2006
16 years ago4,068 views

this is an interestin cache by google

http://72.14.207.104/search?q=cache:R2NvTevo1pcJ:www.microsoft.com/robots.txt+&hl=en&gl=us&ct=clnk&cd=1

TOMHTML [PersonRank 10]

16 years ago #

Just someone who made a link to this file, and google follow the link?

/pd [PersonRank 10]

16 years ago #

not sure, Tom .. but why would google cache the robot.txt ? if you notice this was crawled under the msn domain too and this brings me to the next question, why would anyone make a link (crawlable) to the rob.txt file ??

Just does not make sense

Piotr Konieczny [PersonRank 9]

16 years ago #

I agree with TOMHTML.

pd: some people make links, f.e. to Google's robots.txt or Whitehouse.com one.

Art-One [PersonRank 10]

16 years ago #

http://www.google.com/search?num=100&hl=en&lr=&newwindow=1&safe=off&q=inurl%3Arobots.txt&btnG=Search

There seems to be a lot of robots.txt indexed & cached...
'Results 1 – 100 of about 70,000 for inurl:robots.txt. (0.56 seconds)'

Ludwik Trammer [PersonRank 10]

16 years ago #

LOL, webmasterworld.com has a blog in the working robots.txt file!
http://www.webmasterworld.com/robots.txt

Philipp Lenssen [PersonRank 10]

16 years ago #

Google also indexes pages that aren't linked from anywhere on the web... if only someone visits the page with the Google Toolbar (PageRank option activated) installed.

Kirby Witmer [PersonRank 10]

16 years ago #

Google even indexes its own robots.txt.

http://72.14.209.104/search?q=cache:4V5KG6DE54gJ:www.google.com/robots.txt+%22google.com/robots.txt%22&hl=en&gl=us&ct=clnk&cd=1

Philipp Lenssen [PersonRank 10]

16 years ago #

> There seems to be a lot of robots.txt indexed & cached...

What happens if you include "Disallow: /robots.txt" into the robots.txt file?

There's an old paradox... there's a barber living on a Greek island, and he's the only barber on that island. He has this rule to only cut the hair of people who don't cut their own hair. Now the barber's hair is growing longer and longer, and he wants to do cut it... but is the barber allowed to cut his own hair?

Ludwik Trammer [PersonRank 10]

16 years ago #

> but is the barber allowed to cut his own hair?

Yes, but only once ;)

Forum home

Advertisement

 
Blog  |  Forum     more >> Archive | Feed | Google's blogs | About
Advertisement

 

This site unofficially covers Google™ and more with some rights reserved. Join our forum!