Then how come Google indexes google.com/reader and displays appropriate title/description for Google Reader's login page: http://www.google.com/search?q=google+reader&hl=en&gl=us |
Ionut – I believe that's because the command in robots.txt is:
Disallow: /reader/
and not
Disallow: /reader
The intention probably was for robots to index www.google.com/reader, but ignore everything inside the /reader/ catalog.
Notice that the result you pointed out uses "www.google.com/reader" URL, while the german result Philipp talks about points to "www.google.de/reader/" (with slash at the end). |
Maybe search engines should be clever and use the data for google.de/reader. I wonder how they decide between google.de/reader and google.de/reader/.
http://www.mattcutts.com/blog/seo-advice-url-canonicalization/ |
That's the best search engine result page I have seem for a long time. |
I'm guessing "Krake" and the kraken from pirates of the caribbean have the same etymology haha?
|