Google Blogoscoped

Forum

Is Sun Sentinel blocking Googlebot from its archive?

beussery [PersonRank 10]

Friday, September 12, 2008
15 years ago2,481 views

Is Sun Sentinel blocking Googlebot from its archive?
http://www.sun-sentinel.com/robots.txt

Philipp Lenssen [PersonRank 10]

15 years ago #

This robots.txt currently contains

------------
User-agent: *
Disallow: /search/
Sitemap: http: //www.sun-sentinel.com/sitemap.xml
------------

So it looks like they don't mention Googlebot, but block any bot from their search, which is perhaps the good etiquette of not allowing search results to be indexed by bots...

beussery [PersonRank 10]

15 years ago #

Right that only blocks "search results" where "search" is in the URL but not articles found via search results. According to the site, stories 30 days or older are archived at another site. I'm finding plenty of old (2001+) stories, even xml versions. Nothing is blocking these.....

Forum home

Advertisement

 
Blog  |  Forum     more >> Archive | Feed | Google's blogs | About
Advertisement

 

This site unofficially covers Google™ and more with some rights reserved. Join our forum!