Friday, July 28, 2006

How Google News Indexes

• Google News editors make a choice which sources to include. Sometimes, they find the sites themselves, and at other times, you can simply ask them and see if they take your site. Google News does not have a policy against blogs anymore.
• Google News sorts every story into a “topic cluster.” For example, if 15 news outlets copy the same Reuters news story (which news sites can license), Google will display them in a cluster in their search results, or their News homepage.
  Of course, stories don’t have to be exactly the same to be matched – but if they are too different, they’ll also not appear in the same group. If you want to stand out in Google News search results, make your article be original, or else you’ll be collapsed into a cluster where you may or may not appear on the first results page.
• Once you are indexed by Google News, to make it onto the Google News homepage, you need to have a story be among the most popular topic clusters in a given section (like Entertainment, US, World, Business...).
  Google gives mostly any site the chance to be top, though behind the scenes, there may or may not be an “authority” rank to make it harder for smaller sites to get into top positions on the frontpage (I don’t know either way).
• What Google News needs to index a story is an overview page of some sort linking to full-page articles. Google News does not parse e.g. a page that links to a story anchored within a page containing multiple articles.
• You don’t need to have every story be indexed by Google News. If you provide their crawler with a special overview page (not your frontpage), you can decide which stories to display on this special page. This way, you can decide to not allow e.g. satire to enter Google News. For blogs, this also means you can make sure you don’t feed their crawler with 1-link posts (which may be relevant to your blog, but which are redundant in Google News).
• Google News honors sites that break the news. When you’re the first article to be indexed for a given topic cluster, you will guarantee your site a permanent (or semi-permanent) stay on top of that cluster. That means when you cover the new XYZ first, and an hour later, 90 other news sources talk about XYZ, and then someone searches for XYZ, they will see your story on top. (Of course, these are general rules I’m extracting – I don’t know if this always happens.) The cool thing about this is that in Google News, even a smaller indexed blog can compete with, say, CNN.
• Only Googlers know how many news sites Google News indexes. They’re providing us with the “more than 4,500” number (per country) for a long time, even though they’re adding fresh sites to the index. This either means old sites are frequently replaced by new ones, or – probably more likely – Google News now indexes way over 4,500 sites.
• Google News means speedy indexing... your overview page will be polled every few minutes. A given new article will make it into Google News under 10 minutes. However, sometimes a story doesn’t make it all... I’ve heard wild theories about why this is happening (including the suggestion that Google discriminates against anti-Google stories) but for all I know these are technical hiccups, nothing more.
• Google sometimes removes sources on popular request... this happened with white pride site National Vanguard a while ago.

• Google News powers certain News Alerts. As many people subscribe to News Alerts on their own name, or their company name, having a story being indexed in Google News often means the CEO in question will read it. (Also note that sometimes Google News results are listed as so-called oneboxes on top of web search or image search results.)
• If you include an image in your article, Google News may show a thumbnail of it. Google may have reason to prefer certain types of images – say, middle-sized JPG images, as they’re more likely “story” images than a tiny GIF – though I don’t know for sure. Below every thumbnail, the source is listed, even though the source may not appear in the link selection for this specific cluster.

How Google News Indexes by Philipp Lenssen | Comments (18)

>> More posts

Advertisement