Google Blogoscoped

Forum

Does Google Allow Cloaking When They Like the Site?  (View post)

Marek [PersonRank 1]

Wednesday, December 13, 2006
17 years ago18,339 views

Google indexed webmasterworld.com pages at the time when they were publicly available and there's no cloaking going on here in my opinion.
If suddenly all of Google blogoscoped pages fell behind a paywall like webmasterworld.com, they would still be in Google's index but would it be cloaking?

webmasterworld.com likely has no-cache directive for robots which explains why you can't see a cached copy of those pages.

Nathan Jamin [PersonRank 1]

17 years ago #

What results do you get when doin the "php-based cms" search on Google? The first result I get links to a webmaterworld.com thread which I can access and read without having to sign up...

Am I missing something?

Christian Lund [PersonRank 2]

17 years ago #

I had to login to see the page, however once the session was active I could access the page directly without logging in.

I never paid anything for my Webmaster World account and I still have access to the forum, which I guess is similar to some news sources that force you to create a free account.

Piotr Zgodzinski [PersonRank 4]

17 years ago #

in short, the WebmaseterWorld scam works like this:

http://www.google.com/search?q=%22php-based+cms%22

www.webmasterworld.com/forum44/1287.htm

(...)
GET /forum44/1287.htm HTTP/1.1
Host: www.webmasterworld.com
(...)
HTTP/1.x 302 Found
(...)
Location: http://www.webmasterworld.com/login.cgi?status=google.&urrl=http://www.webmasterworld.com/forum44/1287.htm

I'm not sure if it is strictly cloaking, or rather doorway page, but one thing is for sure, it is unfair.

And one more thing. A year or so ago, there was a simple rule to overcome this deceptive redirect – switch your UserAgent to GoogleBot. It worked briliiantly, directing me to the content, not to the pay-login page.

But sadly it is no longer in place. Maybe they are detecting the GoogleBot by IP address now, rather than UserAgent.

Elias Kai [PersonRank 10]

17 years ago #

Philipp, most newspapers including new york times has those cached content where they still show in google serps.
the purpose can be to get more subscribers ?

JohnMu [PersonRank 10]

17 years ago #

It's still cloaking, no matter what, how or why they do it.

Most visitors who hit the site from the search results (not just for the first time, but for many times after that) will not recognize the content as shown in the search results and will assume that you have to pay for it.

Get over it people. This is cloaking.

The issue of course is not really WMW, it's large sites like the NYTimes. It's a large question of public relations and money (stock price).

It's a question which I believe Google is working on and I'm sure it won't be an easy decision either way. There needs to be an absolutely clear statement about what is and what isn't allowed – cloaking to grab email addresses or "scams" that try to grab donations for free content just don't seem that kosher to me.

I could release a simple javascript that can be included in web pages that does exactly the same thing. Just plug a line into your pages and you'll have the same "feature" (technically different, but the same for the user and search engines). Perhaps we should just add it to as many sites as we can...

Philipp Lenssen [PersonRank 10]

17 years ago #

> Google indexed webmasterworld.com pages at the time
> when they were publicly available

Marek, do you have some reference to back up this statement, e.g. a statement referring to the date when WMW put up the registration box for threads?

And if that's the case, it will be very easy for Matt to debunk this, and everyone will be happy.

> What results do you get when doin the "php-based cms"
> search on Google?

Nathan, shortly before publishing the article, searching for [php-based cms] I get this Google result...
http://blogoscoped.com/temp/wmw-result/search.html

... and this WebmasterWorld page:
http://blogoscoped.com/temp/wmw-result/page.html

(Of course, if you're already registered and signed in it makes sense that you won't see the login box anymore.)

> Philipp, most newspapers including new york
> times has those cached content where they still show
> in google serps.

Elias, it's possible, though I can't reproduce the NYT cloaking right now anymore, at least not this specific case: http://blogoscoped.com/archive/2006-06-19-n18.html
I can now see "please register" boxes when I go to results from this search [site:select.nytimes.com], but the snippet Google shows is exactly (or less) the snippet that the NYT presents to the human visitor. All fair as far as this result goes.

Ryan [PersonRank 0]

17 years ago #

as far as the WSJ and NYT are concerned, they only require registration for old articles... over 30 days, so it's possible they're cached.

I don't have time right now to find an example, but i've seen this happen to wmw threads as old as 2 or 3 days..

I've actually stopped using WMW, as it frustrates me on a daily basis.

Philipp Lenssen [PersonRank 10]

17 years ago #

Ryan, while possible somehow I find that hard to imagine to be the reason here, as the thread "best PHP based CMS" on WebmasterWorld has been created on April 20, 2005. WMW doesn't seem to allow Google to show their cache so I can't see the crawl date, but wouldn't it be likely that Google, who happened to decide that this thread was worthwhile to rank best on [php-based cms], decide to crawl the page again in over a year?

But (again) if that's the reason, I'm happy if Matt just debunks this.

Utills [PersonRank 10]

17 years ago #

This is a strange one. I did exactly what you said and I actually get the page with the correct contents. I'm not sure why but in the past I have sometimes got the subscribe page you show and sometimes I get the actual contents.

http://utills.googlepages.com/webmasterworld.PNG

Philipp Lenssen [PersonRank 10]

17 years ago #

Did you clear your cookies first?
Do you use Google Web Accelerator?
Which location are you at?

kmike [PersonRank 1]

17 years ago #

I, too, get this particular page with the real content, no login boxes.
No proxies. No account at WMW. With or without cookies, all the same.

Utills [PersonRank 10]

17 years ago #

I'm in the UK. On the computer i'm on I've never been on WebmasterWorld and I connect directly to the net.

Utills [PersonRank 10]

17 years ago #

My instinct is that they look at the ip address and if it is a first time viewer they allow the contents to be dsiplayed. However, if you try and view other sections of the website or wish to read more pages it comes up with the subscriber page.

JohnMu [PersonRank 10]

17 years ago #

That's interesting Utills .... it also looks like your header "Welcome to WebmasterWorld Guest from 87.81.174.56" is in bold wheras mine is normal. Could it be a different "version" of the site ... or perhaps they're even quick to change it now (doubt it)...? Do you have the ShowIP plugin in your Firefox?

Have you looked at what they ask for a "contribution"? $89 / 6 months. I seriously do not want to know how much money they pull in from that page.

They have the nocache attribute in place (like most other cloakers) and yes, they do have current threads indexed, see:
http://www.google.com/search?num=100&q=site%3Awebmasterworld.com+%22uk+results+only%22

Philipp Lenssen [PersonRank 10]

17 years ago #

> My instinct is that they look at the ip address and if it is
> a first time viewer they allow the contents to be dsiplayed.
> However, if you try and view other sections of the website
> or wish to read more pages it comes up with the subscriber
> page.

If that's right, that would mean the Googlebot would also see the same login box on repeat visits (the Google crawler is surely a repeat visitor!), or else it's still cloaking, unless Google comes up with an explanation that is.

Note that at this time I also see the login box when I translate this page with Google ...
http://translate.google.com/translate?u=http%3A%2F%2Fwww.webmasterworld.com%2Fforum44%2F1287.htm&langpair=es%7Cen&hl=en&safe=off&ie=UTF-8&oe=UTF-8&prev=%2Flanguage_tools

... or view it through the external BrowserPool (which could all be repeat visitors, of course).

Note my point is not to prove this post's speculation right (or wrong), but to have an actual answer from Google that either says "correct" or "false", because they have the inside data.

Utills [PersonRank 10]

17 years ago #

I actually agree that it is cloaking. I think Google should not display any results that require you to jump through hoops (even if subscription is free), regardless of the reputation of the website in question. Just because Google know that the New York Times or Webmaster World are not going to have spammy links, they should nevertheless treat them the same as they did to a site like BMW.de.

JohnMu [PersonRank 10]

17 years ago #

Yep, they are doing it on an IP address basis. I tried a few proxies with the same cookies in my browser and some (seldom used) proxies work, and others (used by a lot of people) bring the login screen. I bet they either allow a certain number of visitors (or # of days active) from an IP to get in and put the login screen up afterwards.

I think it's unimportant how they are technically doing it – fact remains that they're doing it on purpose and Google is allowing it on purpose. The only problem is that we do not have a statement from Google either for or against this sort of thing.

How's an honest user supposed to know which sites should be reported for cloaking and which shouldn't? :-)

JohnMu [PersonRank 10]

17 years ago #

PS the Google translator page is getting redirected...

Ryan [PersonRank 0]

17 years ago #

Phillip... I think we're bothing trying to say the same thing... me just not as well as you.

I was trying to tell the person who said "maybe google crawled it before they put in the login restriction" that it's unlikely...

Ryan [PersonRank 0]

17 years ago #

I think Matt needs to jump in and clarify for us on this. We've been discussing it (it seems) for months.

Is it because Google is more useful to users with the results than without it, that even though users have to register they'd still find it useful?

That would make sense for the times, as for certain queries an article may be very useful to a user if they register a free account.

Maybe google should somehow indicate on the search page that these results require free accounts to view. I'd also be all for weighing these search results so that non-login-required sites should be more "useful" than a login required site and thus rank above the account needed version.

While it's technically cloaking, their intent isn't to rank for terms that aren't on the page... it's just to make users give them their email first before they can see it.

This is useful to some users, and a pain in the ass to others. I think weighing the results and indicating to me somehow that i need an account would be best for all users involved.

JohnMu [PersonRank 10]

17 years ago #

>While it's technically cloaking, their intent isn't to rank for terms
>that aren't on the page... it's just to make users give them their
>email first before they can see it.
Ok, so they're scraping email addresses. Is that ok?
Look at the login page again, where would a normal user recognize that the content is available for free? "need to register?" in 9px font?

I think this whole discussion would be moot if they had the login page and a nice text explaining that the contents are free, just click here to see them (or solve this captcha, etc). If it was so obvious that the contents were free then would Matt Cutts really have to do a post on how to log in for free?

Philipp Lenssen [PersonRank 10]

17 years ago #

> I think this whole discussion would be moot
> if they had the login page and a nice text explaining
> that the contents are free, just click here to see
> them

I think even if it was completely obvious and you just needed to do as little as click somewhere to get to the page – which I agree is not the case right now! – ... it'd still be cloaking (if that's what it is!) per Google's Webmaster guidelines. Google's guidelines (rightfully, IMO) do not make distinctions about how easy or hard it is to reach the actual content that was displayed in the result... they just say you shouldn't display different results to the Googlebot than the human visitor.

John Hunter [PersonRank 1]

17 years ago #

It sure seems like cloaking to me. Google returns search results for many sites with links to gateway signing pages (either pay or unpaid) with no way to actual view what Google claims is there. I have wished for years at least Google would allow us to block such results as an option. If Google chooses to allow some cloaked content I don't really mind that if they give me an option to say I don't want any results that are cloaked. Then it would also be nice to turn on certain cloaked sites that I have access to, so that Google returns results from those sites.

RonD [PersonRank 0]

17 years ago #

Nailed it right on the head. I was thinking about yesterday or today starting a flame thread @ DP regarding WMW. Yesterday i was trying to find some CSS specific things and god help me i wanted to rip their throats out. Even if the pages were indexed when WMW was public, google should have started dropping them long ago. This really goes to show that you (still) just have to keep the page open briefly then lock it out to the public & start displaying advertisements. I've really been wondering why google doesn't address WMW (and the like) sites. Ridiculous.

Marek [PersonRank 1]

17 years ago #

Philipp,
No, I don't have anything to back up my guess.
Just like you don't have anything to back up your claims of webmasterworld.com using cloaking.
In my case, however, it should be easy to verify.

JohnMu [PersonRank 10]

17 years ago #

>In my case, however, it should be easy to verify.
Search for any fairly recent thread – you'll find it in Google, and you'll meet the login screen when you click on it. Try the example I posted above with "uk results only" + the site:-query.

AN [PersonRank 3]

17 years ago #

Getting a log-in screen from a SERP is extremely annoying, no matter how valuable the content supposedly is, no matter if it's WMW or NYT. I wish Google would throw them all out of the index.

Niraj Sanghvi [PersonRank 10]

17 years ago #

AN: ...or create a partnership with bugmenot.com ;)

Jarid [PersonRank 1]

17 years ago #

I thought it was well known that Google actually encourages this type of behavior, at least for Google News:
http://www.google.com/support/news_pub/bin/answer.py?answer=40543&topic=8871

<blockquote>
In order to include your news articles in Google News, our crawler needs to be able to access the content on your site. Currently, crawlers can't fill out registration forms, nor do they support cookies. Given that, we need to be able to circumvent your registration page in order to successfully crawl your site.

The easiest way to do this is to configure your webservers to not serve the registration page to our crawlers (when the User-Agent is "Googlebot").
</blockquote>
--Jarid

Ryan [PersonRank 0]

17 years ago #

Reading that page linked above, they say what WMW is a bad user experience.

in the news section, they say they label these as "subscription".

SO the technology exists... do it on real search too.

John Honeck [PersonRank 10]

17 years ago #

Obviously Matt Cutts has had ample opportunity to comment on this in his blog and on this excellent forum, and I would guess is going to continue to ignore the concerns. The only response we've seen from him is (paraphrasing) "..Google is against cloaking.."

I think the only real "official" response I can recall was from Adam Lasnik in this thread of the Google Groups: http://groups.google.com/group/Google_Webmaster_Help-Indexing/browse_thread/thread/a9cb3e53ee09bc17/f5ad66d26faa166d?lnk=st&q=working&rnum=3#f5ad66d26faa166d

Dated November 7th, 2006 Adam states, "On a happier note, my colleagues and I are working on an arrangement
which I think you'll be pleased with... balancing many Webmasters'
interest in requiring community membership or sign-in to content-rich
pages while still showing content in Google's search results. Stay
tuned :) (we'll make an announcement in the Webmaster Central blog) "

I think this would have to be some sort of vetting procedure that Google makes available to all webmasters (Not just ones who are chummy with Google and hold conferences in Vegas) signed up for webmaster tools, central, sitemaps, whatever it is. This vetting would be so they could build some sort of trust with the site owner such that the cloaking isn't used to serve different content, harvest names for profit/spamming, charge for content, etc.

To me the real problem with WMW and NY Times being able to serve users the sign-up/upsell/sign-in page rather than the page the content was used to create the listing in the SERP goes to the heart of one of Google's most redeeming qualities. They don’t allow paid inclusion in their organic listings and are not influenced by you paying $299 to be included in some worthless directory. If you do choose to pay by adwords they are clearly delineated as a sponsored listing. I don’t think Brett Tabke or the NY Times paid money to be allowed to cloak, they did cash in some other currency, and that is access. Access is something that joe-sixpack-site-owner doesn’t have.

Philipp’s headline was perfect, “Does Google Allow Cloaking When They Like the Site?” Sites they like…Does that mean sites that don’t sell the list of names harvested, sites that offer good content, sites that only support democratic candidates? You see the slippery slope we are on.

In the very least Google should afford all webmasters the opportunity to have sign-in/upsell/email-harvesting pages without having to have Matt’s phone number on speed dial or an ad spend of a few million a year. When they do that, they should also warn the surfer that the following content requires registration to view. While they are at that how about a radio button to give me the option to include such sites in my search or relegate cloaked/sign-in sites to their own index. Obviously this wouldn’t be as popular as the regular index and if cloaked sites want to be included in the regular index they’ll have to provide some preview content that is freely accessible by everyone, which is really what they should have been doing all along….instead of using the free one provided by Google called the snippet.

Ryan [PersonRank 0]

17 years ago #

I like the "first hit free" approach and think that WMW should offer it up.

That is, if the referer is google's search result page... show me the result... but then cripple the rest of the site so that it nags me.

Jarid [PersonRank 1]

17 years ago #

Johnweb,
While I agree Google needs to clarify its policy on cloaking and first-click-free, I think you need to cut Matt some slack. The post was posted approximately 8 hours ago. I wouldn't exactly describe that as "ample opportunity" for him to respond. Remember, he does have a day job. ;)

John Honeck [PersonRank 10]

17 years ago #

I'm didn't mean to rip on Matt. He's not answering the question for whatever reason, maybe it's the legal department, maybe they will be making an announcement soon, maybe he's in conflict himself and doesn't want that to shine through, or perhaps he just doesn't know why. No one knows but him.

There have been numerous opportunities on his own blog, for example, see the comments here:

http://www.mattcutts.com/blog/how-to-sign-up-for-webmasterworld/

Or even 15 days ago on this very site:

http://blogoscoped.com/forum/77472.html

I just meant to say I'm not going to hold my breath for an answer today officially from Google or anyone unofficially that works there, but I did point out that Adam has said they are working on a system, until then there are the chosen few who are free to build their subscriber base with the aid of Google.

Philipp Lenssen [PersonRank 10]

17 years ago #

Marek:
> No, I don't have anything to back up my guess.
> Just like you don't have anything to back
> up your claims of webmasterworld.com using cloaking.

I never claimed this; I clearly stated this either as *question* or *rumor*. All I am asking for Google is to verify or falsify it. I also try to repeat in every comment here that while it may look like cloaking, I do not know the definite answer to that, and prefer to have Google to give the answer.

> While I agree Google needs to clarify its
> policy on cloaking and first-click-free,
> I think you need to cut Matt some slack.
> The post was posted approximately 8 hours ago.

No Jarid, the post was created two weeks ago (http://blogoscoped.com/archive/2006-11-28-n23.html), and yes I'm simultaneously trying other Google channels to get answers, too, since that day – it was not my intent to follow-up one post that only raises questions with another post that only raises questions... I was hoping to get some answer for a second post.

But yes, I agree Matt has a lot to do, and maybe he wants to do a very thorough post on it and needs to research some issues. I (and many others, judging from the comments the first post received) will certainly be happy if he finds the time... I prefer answers to speculation.

Eric Enge [PersonRank 1]

17 years ago #

My guess is that Google will never offer up an official policy about cloaking, or any clarification on the matter. I cover this in a post I did as a follow-up to this one. You can see my post at:

http://www.stonetemple.com/blog/?p=91

Basically, I believe they do try to identify the sites that do "good" cloaking. They just are not going to promise anyone that they will do a perfect job at it. So the best rule to offer up is "Don't Cloak".

JohnMu [PersonRank 10]

17 years ago #

>So the best rule to offer up is "Don't Cloak".
or at least ... "Don't cloak and get caught without having greased the gears first."
:-)

Ryan, actually the "first hit free" is something that Vanessa Fox more or less mentioned in Matt's blog (no less) in a guest post: "You could make the entire article available for free to users who access it from external links and then require login for any additional articles."
http://www.mattcutts.com/blog/guest-post-vanessa-fox-on-organic-site-review-session/

However, I assume you mean that all pages are indexed and reachable through the serps but any click to a different page is then blocked (even if it were reachable through the serps directly) – from her text I assume that she means to provide a few free articles and block the rest from the visitors and search engine crawlers. Both approaches are interesting...

Philipp Lenssen [PersonRank 10]

17 years ago #

Yes John, that's also how I understand her post... offer some free and some reg-only articles, but treat both visitors and searchbots the same when you do. (Everything else is cloaking per Google's definition anyway.)

Phil Craven [PersonRank 1]

17 years ago #

Great scott! Why can't people get this right. WMW doesn't cloak, and neither does the New York Times. They use conditional auto-redirecting, which is nothing whatsoever to do with cloaking. They serve the same pages to people and to the engines – nothing to do with cloaking.

I do realise that some newer people in the seo industry don't really know cloaking is, because they weren't around when it was commonplace, and they've picked up misinformation in forums and such, so I'll explain what cloaking is:-

Take a 10 page website as an example. Then make another 10 pages for it – one for every normal page – and design them to rank highly in the search engines. We’ll call these pages, “engine pages”. When a person requests one of the 10 pages, send the normal page. When a search engine requests one of the 10 pages, send its equivalent ‘engine page’. People never see the ‘engine pages’, and search engines never see the normal pages. That’s all there is to cloaking.

If a system doesn't serve different pages to the engines than it does to people, and people never see the engines' pages, and vice versa, then it isn't cloaking.

Please let's stop with all this. We had a lengthy debate about it a few months ago at SearchEngineWatch, and it came up again in Matt Cutts' blog a few days ago, and it's all down to newer people not knowing what cloaking actually is. By "newer people", I mean people who weren't in the seo business before Google came along, when cloaking was commonplace.

When the snippet from a URL in the serps doesn't match what is in the page of that URL, then it could be down to cloaking. But WMW's pages of the URLs DO match the snippets that are in the serps, so it isn't cloaking. It may not be nice to have those listings in the serps, but it isn't cloaking.

JohnMu [PersonRank 10]

17 years ago #

Phil, if it's such a clear cut case, why is there no official statement on this? Why such a lengthy debate in SEW?

>Please let's stop with all this.
Let's not. :-).

Sorry to be a PITA but assuming what you're saying is Google's official statement – there are going to be a lot of other sites suddenly adding registration forms, collecting email addresses, talking people into unnecessary subscriptions, etc. Would that be OK with you? Or would you prefer to have the content on the landing page match the snippet in the serps?

Phil Craven [PersonRank 1]

17 years ago #

What cloaking is has nothing to do with Google – the technique was around, and well understood, long before Google was born. And to the best of my knowledge, Google hasn't made an official statement about what WMW and the NYT do.

Calling it cloaking causes a big problem, because (1) Google talks about cloaking as being spam in their guidelines, and (2) many newer people think that some techniques are cloaking when they aren't – as we've seen. It's irresponsible to post that what WMW does is cloaking without first finding out what cloaking actually is.

I can point to a number of people who have been around a very long time – long before Google came to the fore, when cloaking was commonplace – who say that what WMW is not cloaking. One is the main man himself, Danny Sullivan, another is Doug Heil, who is Mr. Whitehat incarnate – and if WMW was cloaking he would shout it from the rooftops, and another is me.

I don't know why it's so hard to grasp. I've explained what cloaking actually is, and always was, and nobody can suggest that what WMW does remotely resembles it.

Phil Craven [PersonRank 1]

17 years ago #

>there are going to be a lot of other sites suddenly adding registration forms, collecting email addresses, talking people into unnecessary subscriptions, etc. Would that be OK with you? Or would you prefer to have the content on the landing page match the snippet in the serps?<

Sorry – I forgot about that part.

I already posted what I think of those systems in Matt's blog, and I've only postde about "cloaking" here. But I'll answer...

I don't like getting stopped by a login/register page, but I prefer that those listings are in the serps, so that I know the content exists, and I can choose to view it or not. I'd rather that paid-only (money) stuff were not in the serps. I accept that my view of the free (not money) content is probably a minority view, but all views are right for the people who hold them.

Incidentally, the "cloaking" issue isn't a matter of my opinion. It's fact.

Eric Enge [PersonRank 1]

17 years ago #

Phil is correct. It is not cloaking unless you serve different content to spiders than users. There are many techniques that may similar to cloaking, but they are not. For example, if you serve different content to a user agent that does not accept cookies, as opposed to one that does.

Phil, since you are familiar with it, can you spell out what type of conditional auto-redirecting WMW does?

Phil Craven [PersonRank 1]

17 years ago #

Hi Eric.

I haven't looked at it in detail, although I've been stopped by the login page a number of times, but I imagine that it first checks if the requestor has a cookie and if they log in automatically, in which case the requested page would be returned. If not, an auto-redirecting page or header will be returned, that causes the browser to fetch the login/register page.

Search engine spiders will be detected, and allowed through. I know that Matt Cutts wrote not to do anything special for Googlebot, but he doesn't mean it to be taken literally. For instance, he *wants* us to use rel="nofollow" specially for Googlebot when appropriate.

I think that one misunderstanding about cloaking has come about by the use of the phrase "poor man's cloaking", being applied to things like hidden text and auto-redirecting. Some people can mistakenly think that those things are a form of cloaking, but what the phrase has always meant is that it is an alternative to cloaking for people who don't want to pay for a cloaking system.

AN [PersonRank 3]

17 years ago #

Phil Craven wrote:

>They serve the same pages to people and to the engines – nothing to do with >cloaking.

They seem to serve Googlebot actual content, while I only get a log-in screen. I don't want to have to register. There's the difference. Then again, I'm using a proxy, but that shouldn't be a problem.

Phil Craven [PersonRank 1]

17 years ago #

>This is a strange one. I did exactly what you said and I actually get the page with the correct contents. I'm not sure why but in the past I have sometimes got the subscribe page you show and sometimes I get the actual contents. <

Isn't this because WMW only puts the Google forum behind a login? I'm sure that's what they used to do, anyway.

Phil Craven [PersonRank 1]

17 years ago #

>They seem to serve Googlebot actual content, while I only get a log-in screen. I don't want to have to register. There's the difference. Then again, I'm using a proxy, but that shouldn't be a problem.<

Yes, but if you register and go to the same URL that is listed in the serps, you will find that both you and Googlebot get the same page. Therefore, it isn't cloaking.

I'm not suggesting that what happens is a desirable thing – only that it isn't "cloaking".

John Honeck [PersonRank 10]

17 years ago #

Call it cloaking, don't call it cloaking, conditional redirect, whatever, the point is the page that is shown in the SERPS is not always the page that the person who clicks it gets sent to. He/she may get there eventually, which is also not the point, it's that extra step in the middle that is the problem. I love the idea of a "subscription required" tag on the result, and of course this comes after all of the non-subscription pages are shown.

For you history buffs, the beginning of this mess can be read up on here:

http://www.webmasterworld.com/forum9/9593.htm

And Here:

http://www.webmasterworld.com/forum9/9618.htm

Phil Craven [PersonRank 1]

17 years ago #

>Why such a lengthy debate in SEW? <

JohnMu: I'll tell you the reason...

It's because the discussion included plenty of people who weren't around when cloaking was commonplace, and they voiced their thoughts as they tried to understand what cloaking is. They were trying to decide whether or not the descriptions of cloaking fitted their thinking, which was no doubt influenced by misinformation.

Google became popular in 2000, and content-based engines (the previous major engines) went into decline, making cloaking largely unnecessary. The seo population these days mostly consists of people who came into it after that, and they never really understood what cloaking was. Through the years since then, plenty of misinformation has been posted in forums and blogs, such as this one, and it becomes more difficult for people to take the word of people who were in it back then. That's why the SEW thread was so lengthy.

John Honeck [PersonRank 10]

17 years ago #

PhilC, perhaps the title of this post should be changed then, but I still think people's concerns and the desire for a definitive answer of if this practice is okay for all sites, not just WMW are valid points, regardless what the technique is called. I really think that is the reason Philipp started this and the other post and is working his magic through other channels.

Even if we ignore the webmaster guidelines that was quoted earlier stating cloaking was not allowed, because we agree that this is not the same as cloaking back in the late 1880s, the spam report for google still has this little check box that states, "Page does not match Google’s description”. And I really doubt WMW would be getting as much search traffic if the log-in page was the one cached during the crawl.

Phil Craven [PersonRank 1]

17 years ago #

I know about the "Page does not match Google’s description” checkbox, and since the page content at the URL that is listed in the serps *does* match the snippet, it would be the wrong checkbox to use.

Yes, the Title of this post should be changed, because it spreads misinformation, which can only confuse people unnecessarily.

I agree that we should have a definitive answer from Google on the issue, but they've known about people's concerns for long enough, and it doesn't seem like they want to comment. My guess is that they wouldn't like it at all if loads of smaller sites did the same thing, but they can't say that in public, so they just keep tight lipped about it. Nothing new there.

justin [PersonRank 0]

17 years ago #

webmasterworld.com results are really starting to annoy the hell out of me – they regularly turn up in php and mysql related searches. if webmasterworld is a walled garden of content then they shouldn't be in Google in the first place.

Phil Craven [PersonRank 1]

17 years ago #

Click on the listings. I think it's only the Google forum that's behind a login.

JohnMu [PersonRank 10]

17 years ago #

It's not just the Google forum.
What purpose would it serve to just "conditional auto-redirect" (I'll stick to "subscription cloaking") the Google forum? That would be really strange :-).

I think your comment sums it up:
>they wouldn't like it at all if loads of smaller sites did the
>same thing, but they can't say that in public, so they
>just keep tight lipped about it.

They allow "it" when they like the site.

Philipp Lenssen [PersonRank 10]

17 years ago #

Phil Craven says

> so I'll explain what cloaking is

It doesn't really matter in this context – I went by the Google Webmaster Guidelines' definition, as I said over and over, and embolded in the post. This case, if anything, is about Google not treating sites equal (if the rumor turns out to be right), so the case is about how Google *tells* webmasters to behave and then about how Google *allows* some sites to behave *instead* of their guideline. And I highlighted their guideline in yellow in the post. "Don't ... present different content to search engines than you display to users, which is commonly referred to as 'cloaking.'"

> When the snippet from a URL in the serps doesn't
> match what is in the page of that URL, then it
> could be down to cloaking. But WMW's pages of
> the URLs DO match the snippets that are in the
> serps, so it isn't cloaking.

I see your point, and I think we're finding out more and more about this case thanks to everyone commenting here, but I disagree about your conclusion. Normal users (on some IPs, when they're not logged in etc.) get redirected away from what they clicked on, so they see something else than what they saw in the snippet. Does Google get redirected too? No, I don't think so, so Google is presented something different. (Sure, you can say, "Googlebot just doesn't get redirected because it doesn't understand client-side redirects, JS or cookies or whatever so it's Google's fault". But by that argument almost any spam tactic is suddenly "OK", e.g. you can start to hide text by printing it white-on-white, too, because technically the text is invisible, it's just Google's fault that Google "sees" it.) And while sometimes a client-side redirect, JS, cookies thing, conditional server-side redirect – or whatever it is that WMW uses – is harmless, in this case it's as clear as it gets that it's intended to put up a login-box, and we know WMW is Google-savvy so they didn't just accidentally do this.

Now while you may consider this "soft cloaking" or whatever, it's only a technicality that's different, and one that doesn't matter to end users. But OK, if you think this technicality is important, and you think Google thinks too – again, we're waiting for a statement here! – then now the world has an officially sanctioned blueprint on how to do something-that-works-like-cloaking-but-is-OK. I'll expect white-hat templates, tutorials, server modules and more to please make this thing open source for everyone, because a lot of sites would just love that. I just expect people to escape to other engines if that ever becomes the case, because it would make Google search results annoying as hell.

Phil Craven [PersonRank 1]

17 years ago #

>What purpose would it serve to just "conditional auto-redirect" ... the Google forum? That would be really strange<

Not really. The Google forum is the one where most of the site's traffic is, so there is sense in restricting it to people who are logged in. Go to WMW and wander round other forums without logging in. There are even a couple of links in this thread that go straight to their thread pages without logging in.

Philipp:

You are right that Google's guidelines state, "Don't ... present different content to search engines than you display to users, which is commonly referred to as 'cloaking.'", but you are mistaken in thinking that the WMW pages contain different content to the what was presented to googlebot and is listed in the serps. Your whole original post is based on that error, and it helps to spread misunderstanding around the seo community. You do the seo community a great disservice by describing what WMW does as cloaking – I explained why above. And it IS a great disservice to the seo community.

I assume you are new yourself, and never understood what cloaking is, so do some research. Listen to the people who have been in this game long enough to know – Danny Sullivan (the main man outside the engines), Doug Heil (Mr. Whitehat), Fantomaster (the master cloaker), and people like me who have been in this game a very long time. Not researching what you are writing about is irresponsible.

>Now while you may consider this "soft cloaking" or whatever, it's only a technicality that's different, and one that doesn't matter to end users<

I don't see it as ANY kind of cloaking – it's not "soft cloaking" or any other kind of cloaking. Cloaking is cloaking, and that's all. Yes, it matters to end users, and I have no argument against objecting to it, but I do argue against the use of the word "cloaking", or even a form of "cloaking", to describe it, because (1) Google talks about cloaking as being spam in their guidelines, and (2) it causes many newer people think that some techniques are cloaking when they aren't.

If you want to write articles about it, call it what it is – conditional auto-redirecting. Please stop trying to redefine things. People have tried and failed in the past. We already have perfectly good words for all the techniques, and it's better for everyone if things don't get confused.

Phil Craven [PersonRank 1]

17 years ago #

If the people I mentioned aren't enough, how about:-

Detlev Johnson and Shari Thurow (two very well-known SEOs who go back a long way):-

"Cloaking is a search engine optimization strategy in which a web page is created for each of the major search engines for top positioning, and only the search engine spiders see the optimized pages. The optimized web pages are never seen by end users. The end users view a different page, hence the term "cloaking."

The article was written in 2000, when Google hadn't become popular and wasn't mentioned, even though other engines are. It can be found at:-

http://www.clickz.com/showPage.html?page=826731

or how about Alan Perkins:-

"IP Cloaking

The identification of search engine robots by IP name or address and the delivery of unique content to those robots. ... the unique content is designed only for the search engine robot to see, not for humans. "

The full article is at:-

http://www.silverdisc.co.uk/articles/spam-classification/#IPCLOAKDEF

It's all out there. WMW doesn't do cloaking.

Philipp Lenssen [PersonRank 10]

17 years ago #

Phil:

> I assume you are new yourself, and never
> understood what cloaking is

I won SEO contests before, but you are right, I never used cloaking :)

> Please stop trying to redefine things.

I don't. Google is being presented something different than end users, and no matter what technicalities are at work, this is/ may be cloaking by Google's definition, and I didn't discuss any other definition in my post (as usual I'm saying "may be" because – as stated in the post and every comment above – I won't reach a final conclusion until I hear Google's official statement on this... maybe this all has a perfectly moral explanation, I'm just afraid I haven't heard it yet). And you may be perfectly right that Google's definition isn't the best, but – and now I'm repeating myself, as I said this above already – what I care about is *Google not adhering to its own guidelines* and letting some sites get away with more than they'd allow your average known, big time V1agra-selling-website.

That being said, the definitions you present here from years ago, in a way, are only further proof that some webmasters simply get creative with a new form of cloaking, so defining the term as plain and simple as Google does in their guidelines seems to be a rather stable definiton... independent of new methodologies, which may pop up over the years, it goes at the heart of the annoyance; "don't let the end user see something else than the searchbot." And that's exactly what happens here: the Googlebot, judging from the snippet, saw "i'm looking to host a new website using a CMS", and when I click on the result, I see "Subscribe to WebmasterWorld, thank you for your support"... *not* quite the same page.

But again, and again, let's have Matt Cutts or someone else at Google react at it. Let them say "it's not cloaking, it's not spam, here's why, and it's perfectly fine to do it, 'cause it's conditional auto-redirecting, totally different, and there's nothing in our guidelines that says you can't do this redirecting, and here's a definition from 2000 which says it's not cloaking, and here's how WMW does it, and it's all cool"... and *then*, when the V1agra-selling site does the same – you'll see a snippet that's *not* available anywhere on the page you end up at, but instead, you'll have a registration box – and the NYT, and the Washington Post, and some blogs do the same, please don't ban any of them, don't do anything against it, same rules for everyone... and *then* I believe that Google treats all sites the same, neutral, without applying double standards. (By then I'd likely escape from Google search for another engine, by the way, because the mildly annoying thing that luckily only happens for a limited result set right now would happen in much larger quantity.)

PS: I'm happy for the point of view you added to the discussion. Please understand I still want to wait for something "official". That's the purpose of Matt's debunking posts, right? Give us a good answer why this rumor is wrong Matt, and we'll all be happy!

Phil Craven [PersonRank 1]

17 years ago #

> Google is being presented something different than end users

No they are not. Go to the URL that is listed in the serps, no matter how you get there, and you'll see that both people and googlebot get the same page.

> this is/ may be cloaking by Google's definition

It isn't cloaking by the definition in their guidelines, and it isn't for us to invent definitions for them.

> what I care about is *Google not adhering to its own guidelines* and letting some sites get away with more than they'd allow your average known, big time V1agra-selling-website.

Google is adhering to their own guidelines in this case, as has been clearly shown and proven in this thread, in the SEW thread, in Matt's blog, and in other places.

>That being said, the definitions you present here from years ago, in a way, are only further proof that some webmasters simply get creative with a new form of cloaking

There's nothing new or creative about the WMW system. It's almost as old as search engines, and a lot older than Google. It has a name – conditional auto-redirecting – that's almost as old as search engines, and a lot a lot older than Google.

> you'll see a snippet that's *not* available anywhere on the page you end up at

You are wrong. The snippet IS on the page you end up at if you, as a person, satisfy the condition.

All you have done in the original post is sensationalise what WMW does, by erroneously using the word "cloaking". You haven't addressed the reality at all. Cloaking is seen as the worst form of search engine spam, and using it in a post like yours is irresponsible, because it is so untrue, and serves only to sensationalise your post.

I have provided plenty of prrof of what cloaking is, and what it always was (have a look at Wikipedia, if I haven't provided enough proof for you), and nobody has come up with anything to show that what WMW does even remotely resembles cloaking.

You weren't around in the search engine business when cloaking was commonplace, so it's understandable that you didn't get to know what it is, and that you've been misinformed. What isn't understandable is that you defend your article in spite of the proof that you have seen (and there's lots more that's very easy to find if you care to look for it). Not only is it not understandable, but it is irresponsible not to change the post, because you are now knowingly confusing newer people in the seo business – presumably for the sake of a bit of sensationalism.

Philipp Lenssen [PersonRank 10]

17 years ago #

Phil, I presented this as a *rumor* that I want to see officially *debunked*. If someone claims something else, I'd say that's sensationalist on their part. Now, as soon as the answer by Matt (or anyone else official) is in, expect there to be a follow-up post where I'll try my best to clarify all lose ends, and I also make sure to include all sides of the discussions, and also your caveats. And as usual, I'll link to the new post from the old.

If no one from Google answers in a couple of weeks, then I might just go ahead and post on this again, but the post would again end in a question mark, followed by a lot of opinions and speculation, followed by a lot of arguments about definitions, and I don't really see how that's very useful to many readers (it may simply be necessary by then, just like I felt the second post on this – the one we're discussing – was necessary, though not very useful, as its title ended in a question mark). See, I'd rather wait for Google to tell me "it's not cloaking, and here's why", because they sit on all the data, and it was their webmaster guidelines that I was basing this post on: <<Make pages for users, not for search engines. Don't deceive your users or present different content to search engines than you display to users, which is commonly referred to as "cloaking.">> And that's exactly what happens to a default user here... the one who you portray as just not happening to "satisfy the condition" – well duh, I dont want to satisfy any condition, by default I want to see the page that was presented to me in the snippet.

Phil Craven [PersonRank 1]

17 years ago #

I've just re-read your original post, Philipp, and it is clear that it accepts that WMW is cloaking. For instance, the Title doesn't ask if WMW is cloaking – it asks if Google accepts the cloaking when they like the site. The basic assumption is that the site cloaks. The whole post talks about it as though it is cloaking. You even state that it is against Google's guidelines when it isn't.

You're asking the wrong question of Google:-

"Does Google Allow Cloaking When They Like the Site [WMW]?"

It's an invalid question because the problem doesn't exist. *A* problem exists, but not that one.

You should be asking why Google allows listings of pages in the serps that many people cannot go directly to. You can kick up a fuss about that if you like, but don't go asking questions about things that don't exist.

Ok. Let me ask you to do something...

Now that you know that WMW doesn't use cloaking (because you now know what cloaking is – right?), and now that you know exactly what technique WMW uses, show me (and anyone else who reads this) exactly which part of Google's guidelines WMW is breaking by doing what they do. And explain (technically) why you think that what WMW breaks that particular guideline.

Incidentally, Google doesn't need pages from the WMW site in the index. They needed the BMW site, and they need the New York Times site, but they don't need the WMW site. It's only a forum, and there are plenty of other forums just like it. It's not a well-known brand like BMW and the NYT are, and they don't need it in the index. That should tell you a lot.

Philipp Lenssen [PersonRank 10]

17 years ago #

I don't know how much Google needs BMW.de or how much they need WebmasterWorld, Phil. All I know is that WMW is no unknown site to the Google Webspam team. Maybe you are right and Google doesn't care about WMW, then Matt can explain this, and the other points as well.

> You're asking the wrong question of Google

For the full picture, please also see the previous post on this in which I asked "WebmasterWorld Cloaking?" and "Isn’t that cloaking?":
http://blogoscoped.com/archive/2006-11-28-n23.html

Again, as for your question, the crucial point to me is the passage*, "Don't ... present different content to search engines than you display to users". So let me ask you a question: do you believe that WMW does this, or do you believe they don't? It's not a rhetorical question – I have a suspicion we may disagree on a more basic point, which is why I'm asking. My answer to this question, by the way, is "maybe, or at least it ends up looking like this – which is the same to the end user – but let's ask Google."

I have another question for you: Do you think the WMW redirect is "sneaky"? Remember, Google's webmaster guidelines* say, "Don't employ cloaking or sneaky redirects", and you say, "there's nothing in our guidelines that says you can't do this redirecting".
Again, no rhetorical question – I dunno if you think WMW's redirects are sneaky, so I'm hoping for your honest answer.
I'll give you my answer to that question too: "I don't know, It might be sneaky... I wonder what Google will tell me about this, so I can make up my mind."

* http://www.google.com/support/webmasters/bin/answer.py?answer=35769

Phil Craven [PersonRank 1]

17 years ago #

Fair enough about the previous post that asked if it is cloaking. I haven't read the comments, so I don't know what was written in them, but I'd guess that people were saying that it's cloaking. If they did, then it's only because the majority of people in the seo business these days are of the newer kind, and don't understand the cloaking technique.

The guideline you think is crossed by WMW is...

> "Don't ... present different content to search engines than you display to users".

The page of the URL that people see has the same content as what googlebot sees, and the snippet shows it. I've shown what cloaking is, so that guideline doesn't apply.

In the old days, there were a lot of pages that were designed to get high rankings for one topic, but when people clicked on the listing, the page auto-redirected them to an entirely different topic, such as porn. (I understand that they still exist). That's auto-redirecting. The page that is listed in the serps does exist and the content is the same for people as it is for spiders, but the auto-redirecting prevented people from actually seeing it. It wasn't cloaking – it was auto-redirecting. Google's guidelines talk about it as "sneaky redirects". That's what sites like WMW and the NYT do, but instead of redirecting all people, they redirect only some people.

> So let me ask you a question: do you believe that WMW does this, or do you believe they don't? It's not a rhetorical question – I have a suspicion we may disagree on a more basic point, which is why I'm asking. My answer to this question, by the way, is "maybe, or at least it ends up looking like this – which is the same to the end user <

No. WMW doesn't do it. They redirect, which is something completely different. I agree that it looks like the same thing to some users, but that's not the point. If you call it cloaking, you have to call all auto-redirecting cloaking, and tell people that all auto-redirecting is spam, because cloaking is spam. But all auto-redirecting isn't spam, as I'm sure you know.

> I have another question for you: Do you think the WMW redirect is "sneaky"? Remember, Google's webmaster guidelines* say, "Don't employ cloaking or sneaky redirects", and you say, "there's nothing in our guidelines that says you can't do this redirecting".<

Personally, I don't think it is what Google means by "sneaky", but that's only my interpretation of their guidelines. Imo, "sneaky" means redirecting to something like I mentioned earlier – a completely different topic, or a different site, etc.

I have a site in which every page has a conditional auto-redirect in it. A while ago I discussed the site with Matt. He looked at it in some detail, and found no fault with the suto-redirects, because they are all needed for the proper functioning of the site. I.e. they aren't "sneaky". Some time later, I discussed the "sneaky redirect" guideline in Matt's blog, and there were people there who simply wouldn't accept that there are non-spam auto-redirects. The reason they wouldn't accept it is because of that "sneaky redirects" guideline. They assume it means ALL redirects. And this is the problem with using the word "cloaking" for what WMW does. There are people who see that "cloaking" is spam in the guidelines, so anything that is called cloaking must be spam as far as they are concerned. That's why I've been discussing the incorrect use of the word "cloaking" here.

I may be wrong, but I don't think that Matt will respond – at least not without the go-ahead from someone higher up. The reason I say that is because he was asked about it in his blog a few days ago, and he chose not to reply to it.

Remember that he is in a difficult position. It doesn't matter what he himself thinks – he can't state his opinions if it would show Google in a bad light. If he says that what WMW does is fine, there would be many sites jump to do exactly the same thing, and he (they) may not want that to happen because too much of it is not nice for the users. If he says it isn't fine, then they would have to remove WMW until they stopped doing it.

It may be that they'd rather not have it in the serps, but they don't have any way of programmatically avoiding it (they don't). But since WMW doesn't break any guidelines, they don't remove it. I have no doubt that if WMW were cloaking, or using "sneaky" redirects, that Google would take action. They acted against the BMW site – big brand, and WMW doesn't even have a brand of any note.

I think it's safe to say that Google accepts WMW and other big sites doing it. They may not like lots of sites doing it, because it's a pain for many of Google's users, but they do accept the big ones. So I really don't think that a statement will come out of Google on this issue, or they would already have made it.

Tony Ruscoe [PersonRank 10]

17 years ago #

<< No. WMW doesn't do it. They redirect, which is something completely different. I agree that it looks like the same thing to some users, but that's not the point. If you call it cloaking, you have to call all auto-redirecting cloaking, and tell people that all auto-redirecting is spam, because cloaking is spam. But all auto-redirecting isn't spam, as I'm sure you know. >>

Is all auto-redirecting cloaking? No.
Is all auto-redirecting spam? No.
Is all cloaking spam? No.
Is there a problem here with the way WMW is indexed? Yes.

<< Now that you know that WMW doesn't use cloaking (because you now know what cloaking is – right?), and now that you know exactly what technique WMW uses, show me (and anyone else who reads this) exactly which part of Google's guidelines WMW is breaking by doing what they do. And explain (technically) why you think that what WMW breaks that particular guideline. >>

How about this:

<< Don't deceive your users or present different content to search engines than you display to users, which is commonly referred to as "cloaking." >>

From: http://www.google.com/support/webmasters/bin/answer.py?answer=35769

As far as I can tell, WMW presents different content to Google than what it displays to me. Otherwise, how could it fairly index that content? Fair enough, I could register on WMW but why should I? Google saw the page I wanted. Why can't I see it too?

It's like going to a library and the librarian saying, "The answer to your question is in this building somewhere; you just have to find out where by jumping throught a few hoops first! Please fill in this form..."

You can argue all you like about what cloaking may or may not be, but the fact remains that the way WMW is indexed is not ideal.

JohnMu [PersonRank 10]

17 years ago #

I agree, Phil – Matt is in a difficult situation here.

However, without a comment he is implying that it is ok to do what WMW does.

Whether or not it's the type of cloaking that was popular in the late 90's or not, I think we can agree that nobody here likes it, from a user perspective, right?

In the end it comes down to:
>I think it's safe to say that Google accepts WMW and other
>big sites doing it. They may not like lots of sites doing it,
>because it's a pain for many of Google's users, but they
>do accept the big ones.

Google allows it when they like the site.

Philipp Lenssen [PersonRank 10]

17 years ago #

> No. WMW doesn't do it. They redirect, which is something
> completely different. I agree that it looks like the same
> thing to some users, but that's not the point. If you call
> it cloaking, you have to call all auto-redirecting cloaking,
> and tell people that all auto-redirecting is spam, because
> cloaking is spam. But all auto-redirecting isn't spam, as
> I'm sure you know.

It's a subtle issue, so I agree we shouldn't over-generalize "redirecting is spam" – Phil, as you say, there are many harmless and good ways. And I also agree that Google might well not have any automated way to find out about this. That's why I think it was important here that we have "beyond a reasonable doubt" knowledge that the Google Webspam team understands what WMW is doing, not necessarily because they cherish a brand, but because it's an important site in the industry, and Matt even did as much as tell people how to get around the login box in one of his posts (there's also some indicators, or call them rumors, that a Google employee, potentially Matt himself, is active on WMW, but I don't think we need to get into that in detail).
http://www.mattcutts.com/blog/how-to-sign-up-for-webmasterworld/

So whether or not there is such a thing as "auto-redirecting cloaking", and I'll certainly try my best to wrap up the subtleties you pointed out here when I do a follow-up or update (still waiting for the official statement), I think the key are the following passages:

   "Don't employ ... sneaky redirects"
   "Don't ... present different content to search engines than you display to users"

We discussed the second part in length above, so instead of repeating all our arguments, including the discussion of what's cloaking, let's analyze the first one: "Don't employ ... sneaky redirects". We know what's a redirect, but what exactly is sneaky? Well, in regards to this discussion of WebmasterWorld, we might compress this into the single question for better analysis:

(1) "When visitors click on a certain search result, is it sneaky to redirect them to a page which presents a different content to them than what was in the snippet the search engine saw (even if that doesn't happen to every user)?"

Answer.com's definition for "sneaky" is "So slow, deliberate, and secret as to escape observation". (See http://www.answers.com/sneaky&r=67 . We probably agree that the "slow" part is not applicable in our discussion. Note that Answer.com also defines sneaky as "Furtive; surreptitious", but frankly I don't know those two words at all, I'm no native speaker, so I'll go with the simpler definition.) I think we all know the two key points from our basic understanding of the word, too:

1. "deliberate": sneakiness must include intent (by the one who's doing it), e.g. "I *knew* it was a fake dollar bill I paid with" (sneaky!) vs "I *didn't* know it was a fake dollar bill I paid with" (innocent!)
2. "escape observation": sneakiness must include a surprise result (by the one who it's being done to), e.g. "I was *expecting* to be handed a real dollar bill, but later on it turned out to be fake" (victim to sneakiness!) vs "I knew it was a fake dollar bill but I decided to accept it" (victim to own stupidity!)

(Disclaimer: paying with fake money is *not* a metaphor for what WMW is doing, but a metaphor for *sneakiness*, which comes in many shades.)

So, based on those two parts of sneakiness:

1. Does WebmasterWorld "deliberately" do this?
2. Does it "escape the observation" of many users, i.e. does it end up being a surprise?

... with "this" and "it" being defined in (1) above.

(Sidenote: surprises, mischievous or not, are the opposite of good usability; good usability is all about being presented what you expected before knowing the final result. Often, a "surprise" is the root of annoyance in software/ web sites. IMO, Google's success has a *lot* to do with good usability.)

I can't finally answer the first question on deliberation, though there's a good chance the answer is "yes", as WMW surely reflects a lot on Google issues, and shows great mastery of the subject (see how they turned their robots.txt into a blog). However, I already saw a single argument in this thread which would go against this part. It's what Ryan said earlier on:

<<as far as the WSJ and NYT are concerned, they only require registration for old articles... over 30 days, so it's possible they're cached.>>

That's a pretty fine argument that what the Wall Street Journal and New York Times doing is not *necessarily* deliberate (the chances decrease by some amount)! Is WMW doing this "fading out" of pages? I dunno, most comments here (including yours Phil) indicate they don't "fade out" articles, and I checked the date of that "php-based CMS" post and is was from 2005... so it seems unlikely that "fading out" is indeed the case, but this would've been a very good explanation. (Then we'd end up discussing whether or not "fading out" your articles is cool, but it'd certainly be a different discussion, with a different focus.)

As for the second question, have a look at the original comments which show some people – not all! – were surprised (it escaped them that this snippet would *not* end up showing on the page they'd end up; of course, they find out afterwards, but by then the time has already been wasted!). I'm assuming "innocent until proven guilty", so I only focus on those who say "guilty!", having a basic assumption of WMW's innocence:

Mambo: "I god damn fall for it everytime."
Cw: "I get that all the time when searching for code snippets. it's a waste of time to click that link then find out you need to be a member to even see the snippet that is shown in the google results."
Dan: "I constantly end up clicking these links in search results for all kind of technical stuff. It is VERY annoying. You think you found a thread relating to your issue, when you click you get a screen telling you to pay them. It drives me nuts."
Erik: "[T]hey don't play by the rules, and it annoys those of [us] trying to use Google."
Jedweb: "I too can't stand this practice and my time has been wasted by it on countless occasions."
JohnMoo: "[T]hey act like they have content but send you to a page with all sorts of payment-logos on it instead."
http://blogoscoped.com/forum/77472.html

I don't try to push an "authoritative argument", I dislike those myself mostly. Could well be all those people commenting are misled! Could be they just don't "get" how Google results are supposed to work sometimes! But sharing the same opinion as the people above, I can certainly identify with their reasoning, so my suspicion is that no, we're not all misled.

So, to wrap this up – and I'm away for the next couple of days, so this wrap-up will have to be my only comment till around Monday night/ Tuesday – there's a good chance* that what WMW is doing is deliberate AND that it is a redirect AND that what they're doing is escaping the observation of many of those looking at search results... which would make it a "sneaky redirect", which would make it against the Google guidelines**, which would mean Google knowingly ignores their own guidelines for some sites. And, IMHO, that last bit is the key issue here.

*"There's a good chance" meaning I want to have a statement from Google before I finally make up my mind on this, though we're awfully close to near the "beyond a reasonable doubt" zone if Google continues to decide *not* to comment. But Matt's a busy man for sure, so I think we can wait some more.

**And we already discussed, at length, the other part that may go against the guidelines... so "sneaky redirect" is only 1 out of 2 potential candidates we're talking about now, the other being "presenting different content to search engines than displayed to users".

Phil Craven [PersonRank 1]

17 years ago #

Tony Ruscoe:

> Is all auto-redirecting cloaking? No.

You are almost right, but not quite. NO auto-redirecting is cloaking.

> << Don't deceive your users or present different content to search engines than you display to users, which is commonly referred to as "cloaking." >>

As far as I can tell, WMW presents different content to Google than what it displays to me. <

You, as an individual, don't matter. WMW *doesn't* present different content to Google than they do to users. Don't forget that little word, "users". It's not just a little word, y'know – it has a meaning.

Allow me to remind you – if the WMW site it spamming, it *would* have been dealt with by Google. This topic isn't new. There are no new revelations here. Google has been aware of what WMW does for a long time. Google is aware that the WMW site presents the same content to googlebot is it does to its USERS.

JohnMu:

> However, without a comment he is implying that it is ok to do what WMW does. <

Yes – that's the way I see it. (Please read below about "users")

> Whether or not it's the type of cloaking that was popular in the late 90's or not, I think we can agree that nobody here likes it, from a user perspective, right? <

It isn't any type of cloaking at all. Most people here don't seem to like it, but I prefer the pages to be listed, even though I get stopped by a login page, and I'm sure that WMW's members are perfectly happy with it. So there are plenty of people who don't mind it, and a few people here who do.

Philipp:

This is only a vague recollection, and I've no idea where to look for it, but I seem to recall a Googler, in the form of Matt or Googleguy – I think Googleguy, addressing this question some time ago. His comment was to ask if you really want pages that are only available to registered users to be listed in the serps. There was no suggestion of it being a no-no. But, as I said, the recollection is very vague, and can't be relied on.

>"Don't ... present different content to search engines than you display to users"

We discussed the second part in length above<

Yes we did, but we overlooked one very important point – that word "users". The guideline doesn't suggest that ALL people should receive the same pages. As long as a site's users receive the same content as googlebot, then the guideline is satisfied. WMW's users receive the same content as googlebot, so the site doesn't fall foul of that guideline.

Since that particular guideline is the only one that has been used to support the idea that WMW is cloaking, I think we can now close the cloaking dicussion. It is only a site's USERS that matter when comparing page content with what googlebot receives. Consider that word "users", and you will never think of WMW as cloaking again.

I appreciate that you went to great lengths to explore the word "sneaky", Philipp, but I don't want to get into that sort of detail. I'm only concerned about the misinformation concerning cloaking.

Personally, I don't think the WMW redirect is sneaky in the way that Google means. I am sure that they mean things like redirecting to pages of different topics, such as adult stuff and other things. But that's just my opinion.

Take a forum for instance. You receive an email, informing you that you have a private message waiting for you, so you go to the forum and click to view private messages. But you aren't logged in, so you get a login page instead. It's a conditional auto-redirect. It isn't sneaky, even though it may surprise you at the time because you'd forgotten that you weren't automatically logged in, and everyone is perfectly happy with it. That's all that WMW does. For certain threads, they want their users to be logged in.

Whether or not it's a good idea to have such threads listed in the serps is open to debate, and that should be the topic, imo. Bringing the idea of spam into it doesn't hold water.

Martin [PersonRank 0]

17 years ago #

The WMW Forums should be thrown out of Google. It is a really annoying thing to click on a result that got the position because of page content and get a login mask and not this page content.

They are playing the users, they are playing Google. Google says they put the users first. This is not putting the users first. I do not care about technical hair splitting, I care about the visible end result as seen by the users, and it is just annoying. Google's credo is the user has to get to see the content that got the page the position when he clicks the result. This is the spirit of the "no cloaking rule", and this is not what is happening with WMW, the specifically design the page to not make this happen.

Phil Craven [PersonRank 1]

17 years ago #

Martin:

It isn't for you, me, or anyone else to decide what the spirit of the "no cloaking rule" is. If Google wants to clarify it, then fine, but until they do, we can only go by what the guideline actually says.

Whether or not WMW should be thrown out of Google, is a matter of opinion, and everyone's opinion is right for them. There can be no argument about it unless people imagine that WMW breaks Google's guidelines.

The bottom line is that WMW hasn't been thrown out of Google, even though Google knows all about it, so it's obvious that there is no breach of the guidelines.

Brian M. [PersonRank 10]

17 years ago #

As I have said before, there is nothing wrong with what WMW is doing. Google Scholar simply would not exist if Google did not allow this kind of cloaking.

Philipp Lenssen [PersonRank 10]

17 years ago #

> The bottom line is that WMW hasn't been thrown out
> of Google, even though Google knows all about it,
> so it's obvious that there is no breach of the
> guidelines.

That's an interesting perspective, Phil... :) I could go into great length debating this point, because I heavily disagree with its (purported) logic, but I'll leave it be for now. I received further feedback suggesting that Google may be close to release a statement on this issue, so I'll postpone further posts/ discussions/ debate myself (though please don't let that stop anyone from continuing in this thread in the meantime).

John Honeck [PersonRank 10]

17 years ago #

Adam Lasnik said they were working on this subject in the thread I quoted above, with the little green G official goodness and all:

http://groups.google.com/group/Google_Webmaster_Help-Indexing/browse_thread/thread/a9cb3e53ee09bc17/f5ad66d26faa166d?lnk=st&q=working&rnum=3#f5ad66d26faa166d

Other than that, I do believe the lack of response does send a signal, as it isn't to unusual to get Adam or Matt to comment on other subjects on a regular basis in all sorts of locations. This subject however is avoided other than to say they don't like it, which is a far cry from it's not allowed. Perhaps the title of this should be changed to "Matt Cutts Devours Babies" that seems to work elsewhere.

Phil Craven [PersonRank 1]

17 years ago #

I hope that I've put the idea of these site cloaking (or even spamming) to bed.

I think this is what you referred to, Philipp:-

"On a happier note, my colleagues and I are working on an arrangement
which I think you'll be pleased with... balancing many Webmasters'
interest in requiring community membership or signin to content-rich
pages while still showing content in Google's search results. Stay
tuned :) (we'll make an announcement in the Webmaster Central blog) "

It was written by Adam Lasnik, who works with Matt Cutts, in November in Google Groups. My guess is that they are working on a system where the 'behind closed doors' pages are still crawled and indexed, but are marked in the serps with something like "registration required" and "subscription required". They could do it by making individual arrangements with site that want to do it, but my best guess is that it will be a meta tag to include in each 'behind closed doors' page.

Forum home

Advertisement

 
Blog  |  Forum     more >> Archive | Feed | Google's blogs | About
Advertisement

 

This site unofficially covers Google™ and more with some rights reserved. Join our forum!