I am not a search expert, but where is the a big difference between the MSNSERACH and Google in the robot.txt file . url below for ease of reference
http://search.msn.com/robots.txt
http://www.google.com/robots.txt
Can someone give a short brief or pointer on what these difference mean for crawl patterns ?? |
Both search engines via the robots.txt file disallow search engines to crawl certain content, most importantly search results. Because when a search engine crawls search results, you get endless garbage results. Many search engines which actually do allow their content to be spidered are doing so on purpose to spam the web with their content (which has AdSense stuck on it). Other services Google disallows to crawl:
- Google Groups
- Google News
- Google Catalogues
- Google Images
- Google's mobile search
- Froogle
- Google Book Search
- Google Maps
and more... |
