Google Blogoscoped

Forum

No Crawling for pages

/pd [PersonRank 10]

Friday, May 26, 2006
18 years ago2,542 views

does google honu a similar syntax for "NO CRAWL" for a web page ??

<META NAME="ROBOTS" CONTENT="NOODP">

Philipp Lenssen [PersonRank 10]

18 years ago #

You can use these...
<meta name="robots" content="noarchive,noindex,nofollow" />

Not sure "nocrawl" is reflects any of these... I suppose once the search engine can read the meta declaration it already *did* crawl the page, so maybe this is only possible via robots.txt.

/pd [PersonRank 10]

18 years ago #

yes, exactly, thats what I thought too..

1) dir/<page> would not included in the Robot.txt
2) Metadata is embedded in the page , so its already hit the page and begin the index
3) Does this mean that if I hit a Page with MEtadata of this sort, will the engine be smart enough to backout and also notify that all)prior) index's based on that page needs to be removed for SERP's ??

ok ok..just asking.. I picked up the meta name strategy for single pages off another player strategy..but still not really understanding the SWAT on this!!

Forum home

Advertisement

 
Blog  |  Forum     more >> Archive | Feed | Google's blogs | About
Advertisement

 

This site unofficially covers Google™ and more with some rights reserved. Join our forum!