I can't remember what all the recent additions were – /cricket, /rebang – but they seem to have disappeared from Google.com's robots.txt, which now appears to end thus:
Disallow: /call Disallow: /archivesearch? Disallow: /archivesearch/url Disallow: /archivesearch/advanced_search Disallow: /base/search? Disallow: /base/reportbadoffer Disallow: /base/s2 Disallow: /urchin_test/ Disallow: /movies? Disallow: /codesearch? Disallow: /codesearch/feeds/search? Disallow: /wapsearch? Disallow: /safebrowsing Disallow: /reviews/search? |
They've also removed "/finance" recently... |