Google Blogoscoped

Forum

Google's Update Speed  (View post)

Art-One [PersonRank 10]

Wednesday, July 26, 2006
17 years ago5,332 views

Imagine thag Google is once updated in minutes, what did i say in seconds, then the Google Blogoscoped search feature would be working on all the items that are just added to the threads and everything could be found very easily with a great certitude ;-)

No really, I don't think we can imagine what the world would be like if that could happen...

BTW finding a law like that would be great, but is it realistic? There are a lot more components playing than with Moore's law (where it is basically only science...)

Niraj Sanghvi [PersonRank 10]

17 years ago #

I'm going to say that the amount of time will be cut in half every 10 months. That comes from going from 30 days in 2003 to 2.5 days today. At that rate, in August of next year it will take less than 1 day to index :)

Playing a bit in Excel, I got D=31.9512(0.938931^m) where D is total days to index given m, the number of months after January 2003. Just remember if it comes true, I take credit :)

Roger Browne [PersonRank 10]

17 years ago #

When the time comes that all your documents are stored on Google's servers, the index will be updated instantly whenever your data changes.

t xensen [PersonRank 4]

17 years ago #

If Google updates every minute or so the SEO types will all be in institutions.

Didn't Zeno have some kind of paradox that relates to this?

Andrew Hitchcock [PersonRank 10]

17 years ago #

I'm not sure how often Google spiders my site, but whenever I make a large change (like redesigning the site), I notice it picks it up on all pages very quickly. I wonder if they are constantly spidering all posts, or if they notice that some pages are changed at the same time as others. Most of my site doesn't change very often, so when they notice one that changed, they start spidering the rest that are connected to that page. Hmm, I hope I explained that well.

Also, if they are going to start spidering the average site more than every day, it might be best to set up a pinging system, so that sites can ping them with new entries (sort of like with blogs). They could develop it as part of sitemaps.

Matt Cutts [PersonRank 10]

17 years ago #

BTW, I think I made the point to Om that Google News and Google Blog Search do pretty much real-time indexing. And Gmail, of course. And Google Desktop, for that matter.

Just to be clear, I don't think I said that the entire index is updated every 2-3 days, but if I did then I misspoke. It's true that when an event happens on the web, our index can often pick it up in 1-2 days, and usually even faster. But a typical page in Google's main web index is updated every 2-3 weeks or faster; it's not the case that the entire main web index is updated every 2-3 days.

Corsin Camichel [PersonRank 10]

17 years ago #

Matt
Are you thinking about a ping like service as a trigger for the bots? Like when a page gets updated, it sends Google a notice? The same way blogsearch, technorati ... work?

Matt Cutts [PersonRank 10]

17 years ago #

I believe that blog search already does work with existing ping services, Corsin. I'm not opposed to it, but I think our existing crawl does pretty well.

Ionut Alex. Chitu [PersonRank 10]

17 years ago #

<< Are you thinking about a ping like service as a trigger for the bots? Like when a page gets updated, it sends Google a notice?

You can use Google Sitemaps for that.

iZeitgeist [PersonRank 10]

17 years ago #

How do you like it when Google indexes one article in your blog and not the ones before it.

A search for Google Milk (http://www.google.com/search?hs=A1O&hl=en&lr=&safe=off&client=firefox-a&rls=org.mozilla%3Aen-US%3Aofficial&q=google+milk&btnG=Search) returns my blog, when you see the cache of that result, it has been indexed/retrieved on 25 Jul 2006 00:16:03 GMT.

Now a popular article on my blog was indexed/retrieved on 25 Jul 2006 08:16:52 GMT.

Still the cache of the main blog (not the article) has not been updated. I wonder why.

But I do give Google credit for indexing my site so quickly (only 2 days after I registred the name), I think it's because it's Wordpress, if it were Blogger it would have taken at least two weeks.

Philipp Lenssen [PersonRank 10]

17 years ago #

I think the higher the PageRank and/ or a page's update frequency, the more often Google will crawl it and update it in their index. At least that would make sense.

Philipp Lenssen [PersonRank 10]

17 years ago #

(Updated the post with Matt's comment.)

Matt Cutts [PersonRank 10]

17 years ago #

Thanks, Philipp. I remember back in 2001-ish, someone from another search engine was quoted in the Boston Globe as their company updating the entire index every 7-10 days. And I knew from our tests that some of that other company's index was updated quickly, but their main web index was only updated once a month.

That's why I don't think I misspoke (I remember that incident), but if I did I wanted to clear it up.

Tadeusz Szewczyk [PersonRank 10]

17 years ago #

"I think the higher the PageRank and/ or a page's update frequency, the more often Google will crawl it and update it in their index. At least that would make sense."

As to higher PageRank = more frequent spidering, it's not as true as it once was. The true frequency of updating gets more important.
New sites without PageRank can circumvent the problem of not being indexed by employing Google Sitemaps and of course getting links.
Back in the days a PageRank 7 link meant a deep crawl. Nowadays you can get deep crawled without that using Google Sitemaps only.

You can also include external RSS feeds on your site to get spidered more often.

Forum home

Advertisement

 
Blog  |  Forum     more >> Archive | Feed | Google's blogs | About
Advertisement

 

This site unofficially covers Google™ and more with some rights reserved. Join our forum!