Monday, March 5, 2007

Google, WMW React on Redirected Results Debate

Matt Cutts, head of Google’s web spam team, has followed-up on the debate that erupted around the way WebmasterWorld served pages. In a nutshell, the WebmasterWorld discussion forums – which frequently pop up in Google search results – often served a registration page on a different URL instead of the content the user clicked on in Google (Google’s webmaster guidelines state that you shouldn’t “deceive your users or present different content to search engines than you display to users,” emphasizing that you also shouldn’t use “sneaky redirects”).

Matt now indicates that he investigated on this behavior and discussed it with Brett Tabke, owner of WebmasterWorld, who then implemented code changes around January 8. The common behavior is that you can now see content of WebmasterWorld pages fine the first time you click on the result. Secondary clicks receive a time-out penalty and are “still under the same access rules and restrictions,” as Brett Tabke writes. Nevertheless this is a serious improvement over the previous situation, and it’s great to find out that the feedback wasn’t ignored.

Why did WebmasterWorld sometimes sent registration pages instead of content in the first place? Brett argues that historically, his forums were suffering from such heavy malicious bot activity (like screenscrapers who simply copied WMW content) that he took measurements like sending corrupted data, or redirecting to login pages, to defend against it. Brett says, “Our first and foremost job is to the regular members of the site. I have done everything I can think of to stop the bots from the cable and dsl isp’s. I have even gone so far as to ban entire tlds some of time (china, russia) where heavy botnet activity exists.” This created a situation in which the Googlebot indirectly became a sort of “premium” user to WebmasterWorld because it was IP-whitelisted, and thus – unlike many searchers – wasn’t hit by the “stray bullets” of Brett’s bot defense.

Part of the root problem here, however, is that Brett decided to not send a response that expresses “your IP has been blocked due to suspected bot activity,” as only this would prevent users from having to second-guess Brett’s intent. (Brett argues he doesn’t send that message to not reveal too much to malicious bots, and also, that he wants to at least get some subscriptions out of those who rip off & republish WMW content.) At this time, WebmasterWorld is still sending the wrong response to some requests, and Matt says he only considers the issue closed when WMW consistently returns the same page (“it should happen for every click from a Google search result”). But for many contexts, the new methodology resolves successfully for searchers... and Google is sending a clear message that they do not want to accept misrepresented result snippets, which in the long run serves searchers better as well.

Update: WMW fixed the last remaining problem, now showing a standardized error message to suspected IPs.

Google, WMW React on Redi ... by Philipp Lenssen | Comments (10)

>> More posts

Advertisement