Google Blogoscoped

Forum

Another Tweak to Google Robots.txt File

Gary Price [PersonRank 10]

Thursday, September 21, 2006
17 years ago2,781 views

Along with the Disallow: /urchin_test/
that Philipp post about earlier
http://blogoscoped.com/forum/68275.html

I also noticed that

Disallow: /base/search?

is now on the list.
This could mean that Google Base results pages cannot be crawled.

Philipp Lenssen [PersonRank 10]

17 years ago #

It seems they exclude search results from being crawled, which is always the right thing to do...

Ionut Alex. Chitu [PersonRank 10]

17 years ago #

It seems they don't (or they didn't):
http://googlesystem.blogspot.com/2006/06/search-google-searches.html

Haochi [PersonRank 10]

17 years ago #

[Moved from new thread "Robots.txt Update". -Philipp]

Disallow: /base/search?
Disallow: /urchin_test/

What's up with the Urchin test...New Google Analytics?

Forum home

Advertisement

 
Blog  |  Forum     more >> Archive | Feed | Google's blogs | About
Advertisement

 

This site unofficially covers Google™ and more with some rights reserved. Join our forum!