Google Blogoscoped


Google Spiders and Ranks Blog on SERPs 26 minutes after Debuting [PersonRank 10]

Friday, December 28, 2007
13 years ago5,247 views

You have to marvel at how fast and powerful Google's technology has become.

Remember years ago, how long it took to get a site listed? Remeber waiting around for the monthly updates to see changes in SERPs.

A Techcrunch article debuted 26 minutes ago, and was spidered and incorporated in the organic SERPs, coming up for keywords in the title -
'Google Mafia' ... ...

Rohit Srivastwa [PersonRank 10]

13 years ago #

I have noticed this thing with my Google alerts & blogs
Alert hardly takes 30 min to reach my inbox after i write a post in my blog or here at blogoscoped

TOMHTML [PersonRank 10]

13 years ago # > I don't understand why you open this thread. This exist for months (Universal Search), and there are many example more explicits: a famous French sportswoman have problems recently (she appears naked on really really private pictures...) and I remember that I have seen a blog ranked first for her name, with the caption "6 minutes ago"...

stefan2904 [PersonRank 10]

13 years ago #

hm, yes. i got some posts from my blog indexed in < 30 minutes... [PersonRank 10]

13 years ago # ...

This latest post was spidered and ranked in 15 minutes on the organic SERPs

and in 10 minutes on the blogsearch

This is how fast the technology is evolving

just imagine a couple of years from now – it will be done in almost real time on all the major search engines

/pd [PersonRank 10]

13 years ago #

I agree with Tom's comments. This is not new, but it's getting real faster. I am getting near practically instance info via the RSS blips within my clipbar. A year ago it was like 20 min for it to get aggregated into GDS and now its 1Min!!


Philipp Lenssen [PersonRank 10]

13 years ago #

Beyond mere technological restrictions of how often a spider can crawl the site, there's also the netiquette of "don't pull the same file too often". (I remember with RSS crawlers I set-up that 30 minutes was the way to go, and that if you polled the RSS too often it would return a blocked file for some sites.)
However, I'd love if important bots would spider say an RSS file or the homepage say every minute.
Then again do we want to give this right to every spider or just the Googlebot? If we give it just the Googlebot* (or just the major search engines) it may become an unfair netiquette, if we give it to every spider, the server load may increase quite a bit...

*E.g. via Google webmaster tools; at this time, the Tools -> "Set crawl rate" page exists, but the "Faster" value is ghosted (unavailable to choose)

Juha-Matti Laurio [PersonRank 10]

13 years ago #

Similar experience here, posts from my blog will be appear to the Google index in 15-16 minutes.

Martin Porcheron [PersonRank 10]

13 years ago #

My blog posts take about 20 minutes to appear in Google SERPs which is odd seeing as Google doesn't crawl my site all that often suggesting that Google might be monitoring RSS feeds that it picks up via FeedFetcher and updating sites when its feed updates

Ionut Alex. Chitu [PersonRank 10]

13 years ago #

[put at-character here]Martin:
Yes, that's obvious and it's the best way to monitor when a site updates (dynamically-generated sitemaps are also good). The tricky part comes when you rank the page, but Google probably uses historical data about the site and changes the ranking after a number of minutes/hours when there's more information about backlinks.

Juha-Matti Laurio [PersonRank 10]

13 years ago #

The Google robot doesn't crawl the site itself daily, absolutely not, especially indexing Web pages updated very frequently Google watch RSS feeds.
BTW: List of some Googlebot UA strings: ...

This thread is locked as it's old... but you can create a new thread in the forum. 

Forum home


Blog  |  Forum     more >> Archive | Feed | Google's blogs | About


This site unofficially covers Google™ and more with some rights reserved. Join our forum!