Google Blogoscoped

Forum

Does the Google Toolbar Index URLs?  (View post)

Ionut Alex. Chitu [PersonRank 10]

Sunday, December 10, 2006
11 years ago13,278 views

I tried something similar.

Date: Aug. 15

Test 1: searching URL1

Test 2: adding URL2 using the form

Test 3: visiting URL3 with Google Toolbar (for IE)

Still nothing in Google search.

Philipp Lenssen [PersonRank 10]

11 years ago #

You mean you added URL2 via google.com/addurl/ and it doesn't show up yet?

Ionut Alex. Chitu [PersonRank 10]

11 years ago #

Yes.

URLi (i=1,3) are pages from a site indexed by Google, but don't have any backlinks.

Ringelnatz [PersonRank 1]

11 years ago #

Philipp,

it only works with the Internet Explorer toolbar – and you have to visit the CIA site at least one time. ;-)

Philipp Lenssen [PersonRank 10]

11 years ago #

Hmm, interesting Ionut. By the parameters of the Toolbar experiment setup you've proven that Google's Add URL form doesn't get URLs indexed. So either the experiment is flawed, or Google's Add URL form is :)

Ionut Alex. Chitu [PersonRank 10]

11 years ago #

No, I think Add URL is mostly useful to index a site, not a specific page. Maybe GoogleBot has many other important pages to index and doesn't look at unworthy pages.

Philipp Lenssen [PersonRank 10]

11 years ago #

Yeah, maybe when they say "Only the top-level page from a host is necessary" that means "we'll actually ignore sub-folders"...

AhmedF [PersonRank 0]

11 years ago #

Just to interject – just because a URL didn't get indexed didn't mean GoogleBot didn't come calling.

JohnMu [PersonRank 10]

11 years ago #

Funny, I have a test site running which is getting automated visitors from several IP ranges with Firefox + Googlebar – since July 05 – and it only got indexed on MSN + Yahoo (who track domain registrations). Your Googlebar must be better than my Googlebar, LOL :-)

John

Ryan [PersonRank 0]

11 years ago #

I ran into a similiar weird problem with Adsense.

As we all know, if adsense caches a site recently and then GoogleBot needs a cache, it won't re-get it if it's near the same time...

anyway, while developing a site I was playing with layout options.. all my text was ipsum lorem decum but I had images, titles, and of course, adsense.

The next day I plugged the real text in, submitted the URL to Google, and blogged about the new site on my blog.

Only problem? The google Cache of my site showed ipsom lorem decum... and did that for over a month until I got some other sites to link to it and GoogleBot re-visited.

alek [PersonRank 10]

11 years ago #

Nice experiment – always fun to debunk some of the tin-foil hat conspiracies with a simple test.

Ditto Ahmedf's suggestion that you should check your web server logs – while Google could index the page without looking at the contents (and correctly does IMHO) for blocked pages that are linked as Matt has pointed out in the past), I assume it would at least make an attempt to spider it if it got into the indexing process.

I.e. a suggested update/clarification to your post would that no attempts to block Googlebot were employed (i.e. robots.txt, meta tag, etc.) and that the web server logs showed that nobody except you (and Matt) accessed that non-password protected page.

P.S. Wasn't clear from the referenced thread what the bet/wager was – what do you "owe" Matt? BTW, this was a "rigged" bet in that all Matt had to do was look at the inputs into the indexing engine to see if their was a feed from the toolbar URL's.

Matt Cutts [PersonRank 10]

11 years ago #

Ionut, while we do take pages from the /addurl form and add some of them to the index, we don't guarantee that every page submitted to /addurl will be added to the index; we still have to prioritize. I'm curious--did you submit the /addurl url with or without filling in the captcha?

alek [PersonRank 10]

11 years ago #

BTW, I didn't see any toolbar pagerank on the "hidden page" either – just "dotting another 'i'" for 'ya ... ;-)

Matt Cutts [PersonRank 10]

11 years ago #

BTW, Philipp, I posted my write-up here:
mattcutts.com/blog/debunking-t ...

rodflash [PersonRank 0]

11 years ago #

I really would like to know hot the vote buttons from Google Toolbar interferes in PageRank...

Ionut Alex. Chitu [PersonRank 10]

11 years ago #

<<I'm curious--did you submit the /addurl url with or without filling in the captcha?>>

I didn't know you could submit it without filling the captcha.

Allard [PersonRank 0]

11 years ago #

Maybe they want to do the same as alexa and collecting information about traffic....

Milly [PersonRank 10]

11 years ago #

Philipp, I don't think you've proved that your "suspicion was dead-wrong", nor that Matt's (misnamed) post debunked it. When you started this, I said blogoscoped.com/forum/62329.ht ... :-

"And if it doesn't appear, wouldn't that only prove that the Toolbar doesn't reliably index *all* the URLs it sees? For the sake of speculation, maybe it's just flaky; maybe it only indexes above certain criteria (to compensate for the lack of PR), e.g. any/all of total hits; different GUID'd hits; different IP'd hits; outgoing links; domain 'authority'; etc."

Isn't that the case now?

You won the bet fair and square, and the experiment is interesting and fun, but (one browser; one URL; "several" visits) surely gets nowhere near proving or debunking the hypothesis. As two guys who 'think like a Googler', where's the scientific rigour? ;)

Philipp Lenssen [PersonRank 10]

11 years ago #

You are right Milly this experiment's setup can't find final proof. I tried to suggest this by writing "as far as this experiment was able to find proof for that, and I can’t think of any setup that would yield final proof". In my opinion, getting scientific proof that the Toolbar *doesn't* index URLs is like proving there's no red bananas living on the moon – it's just incredibly hard to negate something like that. (Try it :)). However, as I don't have any evidence that the Toolbar *does* index URLs, I'll believe it doesn't until I see proof that it does.

Milly [PersonRank 10]

11 years ago #

I spotted your qualification, but you also changed your mind in response to the findings, and talked about being "dead-wrong" and "nope, the Google Toolbar won’t index pages" and (last time) "this one is watertight". That's a mixed message which, together with Matt's quasi-official "debunking-toolbar-doesnt-lead-to-page-being-indexed" post, is likely to be widely reported/blogged as credible 'proof' that the premise is wrong. I just think that's a pity, because it's simply not (yet) the case.

Yes, it's hard to prove a negative. But absence of proof isn't proof of absence, and nor is this experiment.

As you (and I) also said, we'd really need Matt (or someone) to get a definitive, on-the-record, not-possible-by-accident-or-design answer from the Toolbar team. In the meantime, we've really only got yet more anecdotal evidence (which, I agree, is all that can be available to those outside Google), but now blessed with some spurious authority.

(FWIW, I don't believe it's true either, at least by design, largely because I imagine someone within Google who knows would have given Matt a heads-up if he'd been wrong so publicly. So I'm relying on anecdotal evidence, too).

As for red bananas, of course they're on Mars, where all the fruit is red, and there's water :)

Bala Arjunan [PersonRank 1]

11 years ago #

And its nothing to do with Google Toolbar, I tried it and even Yahoo behaves in the same manner.

website design india [PersonRank 0]

11 years ago #

yes i agree that toolbar doesn't help your secret pages get indexed

Matt Cutts [PersonRank 10]

11 years ago #

Milly, the point of the experiment was to verify my on-the-record, not-possible-by-design statements that the Google toolbar would not cause a "secret" unlinked page to be crawled/indexed/ranked.

adame [PersonRank 1]

11 years ago #

Nice experiment idea.

I wouldn't have a problem with Google indexing URLs in this way.

Milly [PersonRank 10]

11 years ago #

Though it's moot now, happily, I just don't see how the experiment was capable of verifying (or disproving) anything (bar not 100% indexing efficiency).

But Matt, had you made any such explicit statements anyway?

Not in the original comments here (which started Philipp's quest), where you said only that ...

"I don't believe that part [re the Toolbar] in parentheses is true; let me know if you've got a source for that and I'll go and comment there..".

Not in your debunking post, where you said only that ...

"You sometimes hear people say "I installed the Google Toolbar, and a day later, Google crawled my secret/unlinked page. Clearly installing the Google Toolbar caused that!" Then you'll often see me post and say "No, it didn’t." You'll often see me point to this page that discusses how a page that you think is secret and unlinked can be crawled (hint: our addurl form is one way, referrer leaks is another)." [...] The main thing is that I'm glad an experiment by a smart third party supports what I've been saying for a while."

(Note that you say "it didn't", not "it couldn't" or "it wouldn't", and that the only hyperlinks are to Google pages which describe other spidering possibilities, not Toolbar impossibilities, and not to other, more explicit, posts of yours. And "what I've been saying for a while" is recursively vague).

Of course you may well have said it, in terms, in lots of other places I (and presumably Philipp) haven't seen.

As your first debunking post (about Google buying Adwords) made admirably clear, the key to debunking is sufficient precision, which this one was lacking.

But, no matter, you've said it now, thanks! :)

P.S. Oops, I see I wrongly credited Philipp with winning the bet, above. Congrats *Matt*, who won it fair and square.

Martin [PersonRank 0]

11 years ago #

I am wondering why the hell you just didnt ask Google support whether or not the toolbar indexes pages?

Philipp Lenssen [PersonRank 10]

11 years ago #

> I spotted your qualification, but you also changed
> your mind in response to the findings, and talked
> about being "dead-wrong" and "nope, the Google Toolbar
> won’t index pages" and (last time) "this one is watertight".
> That's a mixed message which, together with Matt's
> quasi-official "debunking-toolbar-doesnt-lead-to-page-
> being-indexed" post, is likely to be widely reported/blogged
> as credible 'proof' that the premise is wrong.

Milly, the experiment was merely as watertight as I was able to humanly make it, which admittedly isn't completely watertight – it's no scientific evidence, and as I mentioned in my post, it's no final proof either. I was trying to figure out how to wrap up my conclusions with the right balance as to not give false impressions into either direction, and think about it, if I'd emphasized more that "this is no final proof!", then people might also have reported elsewhere, "Scandal! No Final Proof Found That Google Toolbar Doesn't Index Secret Pages!" I'm exxagerating, but sometimes a prominently placed "BUT" inverts what you said before. (Take a look at this post to know what I mean – this guy also "lost a bet" if you want:
blog.centraldesktop.com/commen ...)

But I hope with Matt's comments above the issue is settled. :)

EDIT: I changed "Here's the setup of the bet with Matt that proved this" to "... that was meant to prove this, as far as possible".

Philipp Lenssen [PersonRank 10]

11 years ago #

> I am wondering why the hell you just didnt ask Google
> support whether or not the toolbar indexes pages?

It was a fun bet, not less, not more, and hopefully still illuminating. Besides, this way you don't even have to trust Google. :)

Tadeusz Szewczyk [PersonRank 10]

11 years ago #

If you want do display the PageRank value of the sites you visit without sending your personal info to the mothership you might consider using the SearchStatus Firefox extension:
addons.mozilla.org/firefox/321 ...

It shows lots of other info besides that.

There is also the PRGoogleBar without the mothership feature but it's offline right now so maybe it has been discontinued once again. Last time I checked there was a version for FF 1.5 online but it did not work on my machine.

So I use now the non-Google Googlebar extension.

Philipp Lenssen [PersonRank 10]

11 years ago #

Tadeusz, but surely those services also need to contact the Google servers. They can't possibly calculate this value on their own, they need to contact "mothership". Of course, there's no personal info being sent anyway right?

Tadeusz Szewczyk [PersonRank 10]

11 years ago #

Yes, I assume that.

Elias Kai [PersonRank 10]

11 years ago #

I think the easiest and simplest way was add a link on some old sites, go and visit this link or go and do site:www.site. com + name of the new linked site.

A simple relatinships that show the web why you went from a to b and then using google to b.

Milly [PersonRank 10]

11 years ago #

Philipp, I hope you know I wasn't intending to criticise your methodology, which I agree was as tight as possible (and much more interesting for being declared beforehand). It's just that the results, of themselves, could only ... well, I've already laboured that point ad nauseam.

And yes, that other guy's response to 'losing his bet' came across as graceless and grudging, in stark contrast to your own.

Look at it this way, as a result of your experiment, one way or another, an issue which has been widely debated (in such circles) for *years*, now has a firm resolution, for all but the pathologically Google-sceptical. That can't be bad :)

Penders [PersonRank 0]

11 years ago #

But if I was Google, would I index a page that contained just one line of text, starting with: "Will you index me Google?....", hhhmmmm?!

Maybe Google is just implementing a bit of AI, afterall... as AhmedF states above, "...just because a URL didn't get indexed didn't mean GoogleBot didn't come calling".

What if your 'hidden page' actually contained something worth indexing? ... a 'hidden mini site' perhaps?!

Milly [PersonRank 10]

11 years ago #

> [...] sometimes a prominently placed "BUT" inverts
> what you said before.
>
>> And yes, that other guy's response to 'losing his
>> bet' came across as graceless and grudging, in
>> stark contrast to your own.

The 'Net is a small and busy place ;)

mattcutts.com/blog/ie7-promo-p ...

Guillermo [PersonRank 0]

11 years ago #

This "experiment" is biased, because, what if matt cutts decided to block googlebot visiting your secret page in order to win the bet?

Tony Ruscoe [PersonRank 10]

11 years ago #

Guillermo, that would be impossbile since Matt Cutts didn't know the location of the secret page.

This thread is locked as it's old... but you can create a new thread in the forum. 

Forum home

Advertisement

 
Blog  |  Forum     more >> Archive | Feed | Google's blogs | About
Advertisement

 

This site unofficially covers Google™ and more with some rights reserved. Join our forum!