Google Blogoscoped

Forum

Purpose of robots.txt

ALPHAbeta [PersonRank 1]

Tuesday, June 14, 2011
12 years ago5,413 views

Google creates and breaks his own rules.

I am surprised to see this, but what is the purpose of robots.txt then? All Blogger.com blogs have robots.txt file and it blocks /search URLs. But, still URLs appear in Google's index.

Example:

http://googlesystem.blogspot.com/robots.txt

http://www.google.co.in/search?q=site%3Agooglesystem.blogspot.com+inurl%3Asearch%3Fupdated-max

ALPHAbeta [PersonRank 1]

12 years ago #

sorry for grammar mistakes! :)

Forum home

Advertisement

 
Blog  |  Forum     more >> Archive | Feed | Google's blogs | About
Advertisement

 

This site unofficially covers Google™ and more with some rights reserved. Join our forum!