Google Blogoscoped


GoogleBot triggered by Site Search?

FofR [PersonRank 2]

Monday, March 31, 2008
14 years ago2,432 views

Out of curiosity I decided to check the number of pages Google had indexed with a site search (site:). At a time corresponding to this search, the access logs show that the Google-bot cranked up and started indexing the site. This implies that the bot was triggered by my search.

Has anyone else noticed this sort of behaviour?

Roger Browne [PersonRank 10]

14 years ago #

From an engineering point of view, it would make no sense to have search and crawling so closely coupled. Your experience can't be more than co-incidence.

FofR [PersonRank 2]

14 years ago #

I agree, if it was coupled with every "site:" search it'd have all sorts of issues. Though maybe it's just one clause in a set of rules.

"site:" is used on websites that allow users to search their pages using Google. So it makes sense from this perspective that, if they perceive the site to be out of date, they update it to make search results accurate.

I'll run some more "tests" to make sure it's not just a coincidence.

Zim [PersonRank 10]

14 years ago #

It doesn't look crazy. Why not? Lets take two sites, Blogoscoped and some random blog updated once per week. This forum is constantly changing, there's a new comment every hour. To keep up to date the index, the bot will come many times to fetch new pages. But in the other blog, only some times per week.
I'm not sure if it's this way, but I think that's how it works.
With Google Reader I see blogoscoped forum feed is updated often than a friend's blog (updated rarely/less readers).

Colin Colehour [PersonRank 10]

14 years ago #

[put at-character here]Zim, Ask your friend to look into his log files to see just how often googlebot shows up.

Forum home


Blog  |  Forum     more >> Archive | Feed | Google's blogs | About


This site unofficially covers Google™ and more with some rights reserved. Join our forum!