Robot.txt file

[Arleigh]

Basically, the[FONT=Verdana, sans-serif] main function of the robots.txt file is to instruct robots not to crawl and index certain files most particularly password protected folders and folders which contain only images.[/FONT]

 

[temi]

I did a blogpost about robots.txt files here. You can also generate it automatically this days with help from Google via your Google webmaster tools

 

[azseoguy]

Have you found that using the robots.txt file with disallow effectively works to stop your pages from showing in a search index?

Have you ever used noindex in the robots.txt file? I hear it is only accepted by Google.

 

[temi]

Robots.txt is a W3C standard.... it applied to all good crawlers out there so its now used by Google only

 

[Robdale]

Basically, robots.txt file will not do anything to improve your search engine positioning but it indicate robots which files you will not allow to be crawled and indexed in the search engines. Whenever a robot crawls your site it looks for the robots.txt file. If it doesn't find one it assumes automatically that it may crawl and index the entire site.

 

[xhan]

can you use robots.txt to stop google caching certain phrases? I dont want my website to be found by googling my email address!

 

[surreypcsupport]

can you use robots.txt to stop google caching certain phrases? I dont want my website to be found by googling my email address!

No, you can only stop the crawlers from finding pages and therefore they will not be indexed.

 

[azseoguy]

can you use robots.txt to stop google caching certain phrases? I dont want my website to be found by googling my email address!

In the meta tags for that page you could use noarchive. That tells the SE not to cache your web page. I don't think it can be targeted to a single phrase though.

<META content="NOARCHIVE" name="ROBOTS">

 

[xhan]

@surreypcsupport
@azseoguy

cheers guys, i've no archived it - I hope it helps!! :S