Google Blogoscoped

Thursday, March 13, 2008

Updated Google Quality Rater Guidelines

A newer version of what seems to be the guideline document for Google’s quality raters* has surfaced (I don’t know when it surfaced, but it may have been recently). Brian Ussery gave it a look already. According to the document, which is dated April 2007 and at least looks legitimate, a quality rater has the job to first research and understand a specific search query – say [cell phones] –, to then look at the quality of a website returned for this query. The rater will then assign a rating to this specific “query-page” combo, and proceed to the next “query-page” task.

Back in 2005, already discovered information on Google’s quality raters. Note besides quality raters, Google also has trusted testers, but those are different job roles – and the latter is unpaid, last thing I heard.

Quality raters use the evaluation system EWOQ, logging in with their approved Gmail account. (If I understand it right, this system is part of a larger “Rater Hub” homepage. Note none of the two previous links should work for you unless you’re a rater, I’m just including them for reference.) If you want to work in quality evaluation, a Google job page states, you need to have in-depth, “up-to-date familiarity with English-speaking web culture and media,” as well as a broad ranges of interests and strong web research skills.

The quality rating options for a specific URL are: vital, useful, relevant, not relevant, off-topic, didn’t load, foreign language, and unratable. Also, a URL can be flagged as not spam, maybe spam, or spam, and malicious or pornography. We have reason to believe the feedback given by quality raters is then incorporated into the algorithms by Google engineers, or helps deciding with which fine-tuned variant of the algorithms to proceed. (Google once told people their results are “completely objectively,” “completely automated” and “independent of the beliefs” of people working at Google, but they since revised many occurrences of such wording on their help pages.)

As before, queries are grouped into the types navigational (when a user just wants to locate a specific webpage they have in mind, e.g. typing [ibm]), informational (when someone researches a topic to find out more about something, e.g. querying for [tsunami]), and transactional (when the user wants to buy something or download something or is looking for some other type of resource, as opposed to information, e.g. for a query like [download adobe reader]). These types are not mutually exclusive; Google lists the query ["ipod nano"] as an example of something which is navigational (going to the product page), informational (reading a review or info on the product page), as well as transactional (perhaps purchasing the product).

In the document, Google (if it’s by Google, though again, it looks authentic) also describes the different types of spam found online. In their list: pay-per-click pages, JavaScript redirects, parked domains, keyword stuffing, “thin affiliates,” “100% frame,” hidden text/ hidden links, and “sneaky redirects.” Parked domains are an interesting item, as Google themselves makes AdSense bucks from their program... classic case of doublethink. “Thin affiliates,” Google explains, are pages merely existing to push the visitor to another page by another owner, making some commission spare change from this referral. A “100% frame” page, on the other hand, simply wraps another page with its own invisible frameset, thus basically appearing to own the other site’s content.

[Hat tip to Beussery and Vizualbod! Image by Google.]

Update: Yeah, the document is authentic, I just received further confirmation.

*Edit: The document at has been removed by now so I removed the link, though the file is still circulating online in different places in both HTML and PDF versions.


Blog  |  Forum     more >> Archive | Feed | Google's blogs | About


This site unofficially covers Google™ and more with some rights reserved. Join our forum!