does google honu a similar syntax for "NO CRAWL" for a web page ??
<META NAME="ROBOTS" CONTENT="NOODP"> |
You can use these... <meta name="robots" content="noarchive,noindex,nofollow" />
Not sure "nocrawl" is reflects any of these... I suppose once the search engine can read the meta declaration it already *did* crawl the page, so maybe this is only possible via robots.txt. |
yes, exactly, thats what I thought too..
1) dir/<page> would not included in the Robot.txt 2) Metadata is embedded in the page , so its already hit the page and begin the index 3) Does this mean that if I hit a Page with MEtadata of this sort, will the engine be smart enough to backout and also notify that all)prior) index's based on that page needs to be removed for SERP's ??
ok ok..just asking.. I picked up the meta name strategy for single pages off another player strategy..but still not really understanding the SWAT on this!!
|