In crawler speak, this means: stay out of our images directory! Indeed if there’s any site in the world where the owners don’t need to think about SEO in traditional terms, it’s Google.
LIFE images still do appear in Google Image Search, ever since the service was released in November last year. Google’s internal programs can internally access whatever Google stores without looking at any robots.txt.
Asking Google for their reasoning behind barring external image search engines from accessing their site, I received a reply, but it didn’t actually provide a reason:
While Google allows crawling of many of its own properties from Blogger to Knol, the LIFE photo archive is not available for crawling at this time. To learn more about the licensing or merchandising of these images, visit www.timelifepictures.com.
Andy Baio tells me, “It’s disappointing that Google gets exclusive access to index these images and every other search engine is out of luck. Exclusivity like this doesn’t seem in line with Google’s philosophy. To me, it’d be like the Flickr Commons only allowing Yahoo crawlers or Picasa blocking access to anything but Googlebot.”
>> More posts