Google Blogoscoped

Forum

Slimmed back robots.txt

photoactive [PersonRank 10]

Tuesday, March 27, 2007
17 years ago2,426 views

I can't remember what all the recent additions were – /cricket, /rebang – but they seem to have disappeared from Google.com's robots.txt, which now appears to end thus:

Disallow: /call
Disallow: /archivesearch?
Disallow: /archivesearch/url
Disallow: /archivesearch/advanced_search
Disallow: /base/search?
Disallow: /base/reportbadoffer
Disallow: /base/s2
Disallow: /urchin_test/
Disallow: /movies?
Disallow: /codesearch?
Disallow: /codesearch/feeds/search?
Disallow: /wapsearch?
Disallow: /safebrowsing
Disallow: /reviews/search?

Philipp Lenssen [PersonRank 10]

17 years ago #

They've also removed "/finance" recently...

Forum home

Advertisement

 
Blog  |  Forum     more >> Archive | Feed | Google's blogs | About
Advertisement

 

This site unofficially covers Google™ and more with some rights reserved. Join our forum!