Friday, March 2, 2007

Being "the Google" Of Some Niche...

Google has become somewhat synonymous with “leader in a field”, Paul McNamara muses, sort of like “X is the Porsche of Y”... e.g. “Keep your eye on Ubuntu, it just may become the Google of Linux” or “Remember the yellow pages? The Google of the paper era?” [Thanks Paul!]

Being "the Google" Of Som ... by Philipp Lenssen | Comments (1)

How Long Until Google Acts on a Changed Robots.txt?

Alek Komarnitsky did a little searchbot experiment. For a long time, he had a special robots.txt file that commanded search engines to ignore a “bad_robots” directory on his webserver, and he tracked if any of the big search engines would ignore this command (none did). Then, some days ago (on February 10 18th), he removed this command from the robots.txt file to see how quickly search engines would take notice and spider the “bad_robots” directory.

(Spoiler: of the three big search engines Yahoo, Google and MSN, Google was the first to display “bad_robots” in search results, albeit the Googlebot wasn’t the first to index crawl the page.)

Disclaimer: There’s a fair portion of potential coincidence as this is a non-represenative statistical sample... it might well be that another server repeating the experiment comes to completely different results.

[Thanks Alek!]

How Long Until Google Act ... by Philipp Lenssen | Comments (3)

>> More posts

Advertisement