Monday, April 24, 2006

Greedy Google Ad Bot Debunked

Googler Matt Cutts provides an interesting explanation of why it appears that sometimes, the Google AdSense bot indexes pages for Google’s main index, which would be an unfair advantage to webmasters relying on Google ads (see previous post). The truth is yes it does, and no it doesn’t: what happens is that Google implemented a proxy system to cache request. So (from what I understand) when the bot for Google’s main index wants to crawl a page and sees that AdSense already requested it, it can grab the cached copy from Google’s servers, thus saving everyone’s bandwidth. Rules from the robots.txt file still apply separately, Matt says.

Greedy Google Ad Bot Debu ... by Philipp Lenssen | Comments (5)

>> More posts

Advertisement