Is Sun Sentinel blocking Googlebot from its archive? http://www.sun-sentinel.com/robots.txt |
This robots.txt currently contains
------------ User-agent: * Disallow: /search/ Sitemap: http: //www.sun-sentinel.com/sitemap.xml ------------
So it looks like they don't mention Googlebot, but block any bot from their search, which is perhaps the good etiquette of not allowing search results to be indexed by bots... |
Right that only blocks "search results" where "search" is in the URL but not articles found via search results. According to the site, stories 30 days or older are archived at another site. I'm finding plenty of old (2001+) stories, even xml versions. Nothing is blocking these..... |