The Most Active and Friendliest
Affiliate Marketing Community Online!

“AdsEmpire”/  Direct Affiliate

Robot.txt file

Arleigh

New Member
affiliate
Basically, the[FONT=Verdana, sans-serif] main function of the robots.txt file is to instruct robots not to crawl and index certain files most particularly password protected folders and folders which contain only images.[/FONT]
 
Have you found that using the robots.txt file with disallow effectively works to stop your pages from showing in a search index?

Have you ever used noindex in the robots.txt file? I hear it is only accepted by Google.
 
Robots.txt is a W3C standard.... it applied to all good crawlers out there so its now used by Google only
 
Basically, robots.txt file will not do anything to improve your search engine positioning but it indicate robots which files you will not allow to be crawled and indexed in the search engines. Whenever a robot crawls your site it looks for the robots.txt file. If it doesn't find one it assumes automatically that it may crawl and index the entire site.
 
can you use robots.txt to stop google caching certain phrases? I dont want my website to be found by googling my email address!
 
can you use robots.txt to stop google caching certain phrases? I dont want my website to be found by googling my email address!

In the meta tags for that page you could use noarchive. That tells the SE not to cache your web page. I don't think it can be targeted to a single phrase though.

HTML:
<META content="NOARCHIVE" name="ROBOTS">
 
banners
Back