Google has become somewhat synonymous with “leader in a field”, Paul McNamara muses, sort of like “X is the Porsche of Y”... e.g. “Keep your eye on Ubuntu, it just may become the Google of Linux” or “Remember the yellow pages? The Google of the paper era?” [Thanks Paul!]
Alek Komarnitsky did a little searchbot experiment. For a long time, he had a special robots.txt file that commanded search engines to ignore a “bad_robots” directory on his webserver, and he tracked if any of the big search engines would ignore this command (none did). Then, some days ago (on February
10 ), he removed this command from the robots.txt file to see how quickly search engines would take notice and spider the “bad_robots” directory.
(Spoiler: of the three big search engines Yahoo, Google and MSN, Google was the first to display “bad_robots” in search results, albeit the Googlebot wasn’t the first to
index the page.)
Disclaimer: There’s a fair portion of potential coincidence as this is a non-represenative statistical sample... it might well be that another server repeating the experiment comes to completely different results.
>> More posts