Google Blogoscoped

Friday, March 2, 2007

Being "the Google" Of Some Niche...

Google has become somewhat synonymous with “leader in a field”, Paul McNamara muses, sort of like “X is the Porsche of Y”... e.g. “Keep your eye on Ubuntu, it just may become the Google of Linux” or “Remember the yellow pages? The Google of the paper era?” [Thanks Paul!]

How Long Until Google Acts on a Changed Robots.txt?

Alek Komarnitsky did a little searchbot experiment. For a long time, he had a special robots.txt file that commanded search engines to ignore a “bad_robots” directory on his webserver, and he tracked if any of the big search engines would ignore this command (none did). Then, some days ago (on February 10 18th), he removed this command from the robots.txt file to see how quickly search engines would take notice and spider the “bad_robots” directory.

(Spoiler: of the three big search engines Yahoo, Google and MSN, Google was the first to display “bad_robots” in search results, albeit the Googlebot wasn’t the first to index crawl the page.)

Disclaimer: There’s a fair portion of potential coincidence as this is a non-represenative statistical sample... it might well be that another server repeating the experiment comes to completely different results.

[Thanks Alek!]


Blog  |  Forum     more >> Archive | Feed | Google's blogs | About


This site unofficially covers Google™ and more with some rights reserved. Join our forum!