Google’s webmaster guidelines state: “Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don’t add much value for users coming from search engines.”
But guess what: the New York Times – previously in search engine news for offering “spider bites” to the world – not only allows indexing of some of their search results (about 338,000, at this time), violating Google’s guidelines. They’re also doing exceptionally well with this tactic, ranking in the top 5, for instance, for the search query sex.
[Via John Andrews/ Digg.]
>> More posts
Advertisement
This site unofficially covers Google™ and more with some rights reserved. Join our forum!