User-agent: *
Allow: /searchhistory/
Disallow: /news?output=xhtml&
Allow: /news?output=xhtml
Disallow: /search
Disallow: /groups
Disallow: /images
Disallow: /catalogs
Disallow: /catalogues
Disallow: /news
Disallow: /nwshp
Disallow: /?
Disallow: /addurl/image?
Disallow: /pagead/
Disallow: /relpage/
Disallow: /relcontent
Disallow: /sorry/
Disallow: /imgres
Disallow: /keyword/
Disallow: /u/
Disallow: /univ/
Disallow: /cobrand
Disallow: /custom
Disallow: /advanced_group_search
Disallow: /advanced_search
Disallow: /googlesite
Disallow: /preferences
Disallow: /setprefs
Disallow: /swr
Disallow: /url
Disallow: /m?
Disallow: /wml?
Disallow: /xhtml?
Disallow: /xml?
Disallow: /imode?
Disallow: /jsky?
Disallow: /pda?
Disallow: /sprint_xhtml
Disallow: /sprint_wml
Disallow: /pqa
Disallow: /palm
Disallow: /gwt/
Disallow: /purchases
Disallow: /hws
Disallow: /bsd?
Disallow: /linux?
Disallow: /mac?
Disallow: /microsoft?
Disallow: /unclesam?
Disallow: /answers/search?q=
Disallow: /local?
Disallow: /local_url
Disallow: /froogle?
Disallow: /froogle_
Disallow: /print?
Disallow: /books?
Disallow: /scholar?
Disallow: /complete
Disallow: /sponsoredlinks
Disallow: /videosearch?
Disallow: /videopreview?
Disallow: /videoprograminfo?
Disallow: /maps?
Disallow: /translate?
Disallow: /ie?
Disallow: /sms/demo?
Disallow: /katrina?
Disallow: /blogsearch?
Disallow: /blogsearch/
Disallow: /blogsearch_feeds
Disallow: /advanced_blog_search
Disallow: /reader/
Disallow: /uds/
Disallow: /chart?
Disallow: /transit?
Disallow: /mbd?
Disallow: /extern_js/
Disallow: /calendar/feeds/
Disallow: /calendar/ical/
Disallow: /cl2/feeds/
Disallow: /cl2/ical/
Disallow: /coop/directory
Disallow: /coop/manage
Disallow: /trends?
Disallow: /trends/music?
Disallow: /notebook/search?
Disallow: /music
Disallow: /browsersync
Disallow: /call
http://google.com/robots.txt |
<< Google Robots Changes Frequently? >>
I wouldn't say *that* frequently. What's the point you're trying to make? |
see this page
the below page says nothing to see hear which means there is something to see there...Lol
google.com/pagead/
google.com/relpage/
and check this co-brand page
http://www.google.com/cobrand
why the logo in this page of google still in beta ?
http://www.google.com/jsky?
Google Sponsered Link Search
http://www.google.com/sponsoredlinks
see this
http://www.google.com/blogsearch_feeds
a different 404 error from google
http://www.google.com/uds/
Google Bad Request
http://www.google.com/chart?
|
i mean to say that by watching the robots.txt file occasionally one can find new service like ..... the music trends etc........ |
what are you doing Sankar ? ...
note : /UDS/ is the directory where stuff from AJAX search API are stored
|
and by the way what is this page ?
http://www.google.com/errors/purchases/serviceUnavailable.html
|
Yes, those are all URLs in the "robots.txt" file... sorry, but I still don't understand what point you're trying to make.
What did you mean by the "Google Robots Changes Frequently?" title? |
i'm sorry about the title i was just on a hurry.... Sorry tony... |
<< i mean to say that by watching the robots.txt file occasionally one can find new service like ..... >>
I see. That's exactly what I said here:
<< Changes to Google’s robots.txt files
Google try their best to not index their own SERPs, so when they create a new service that could clutter up their index, they’ll usually add a new entry to their robots.txt file. (One of the most recent additions to the file was www.google.com/call, as Garett Rogers discovered.) By scheduling a simple script, you could easily monitor this file and be alerted of any changes. >>
From: http://blogoscoped.com/archive/2006-08-11-n67.html |
Yep. I got a tracker script on Google robots.txt. Unfortunately I was sick during the last alert... Google Music Trends. |
