“webinar”/ “VOLUUM”/
Welcome to Our Community
Wanting to join the rest of our members? Feel free to sign up today.

Robot.txt file

Discussion in 'Search Engine Optimisation' started by Arleigh, May 6, 2009.

  1. Arleigh

    Arleigh Affiliate affiliate

    361
    0
    0
    Basically, the[FONT=Verdana, sans-serif] main function of the robots.txt file is to instruct robots not to crawl and index certain files most particularly password protected folders and folders which contain only images.[/FONT]
     
  2. Voluum
  3. temi

    temi Affiliate affiliate

    13,674
    53
    0
    I did a blogpost about robots.txt files here. You can also generate it automatically this days with help from Google via your Google webmaster tools
     
  4. azseoguy

    azseoguy Affiliate affiliate

    13
    0
    0
    Have you found that using the robots.txt file with disallow effectively works to stop your pages from showing in a search index?

    Have you ever used noindex in the robots.txt file? I hear it is only accepted by Google.
     
  5. temi

    temi Affiliate affiliate

    13,674
    53
    0
    Robots.txt is a W3C standard.... it applied to all good crawlers out there so its now used by Google only
     
  6. Robdale

    Robdale Affiliate affiliate

    470
    2
    0
    Basically, robots.txt file will not do anything to improve your search engine positioning but it indicate robots which files you will not allow to be crawled and indexed in the search engines. Whenever a robot crawls your site it looks for the robots.txt file. If it doesn't find one it assumes automatically that it may crawl and index the entire site.
     
  7. xhan

    xhan <span style="color: white; font-weight: 700; backg affiliate

    591
    10
    0
    can you use robots.txt to stop google caching certain phrases? I dont want my website to be found by googling my email address!
     
  8. surreypcsupport

    surreypcsupport Affiliate affiliate

    584
    3
    0
    No, you can only stop the crawlers from finding pages and therefore they will not be indexed.
     
  9. azseoguy

    azseoguy Affiliate affiliate

    13
    0
    0
    In the meta tags for that page you could use noarchive. That tells the SE not to cache your web page. I don't think it can be targeted to a single phrase though.

    HTML:
    <META content="NOARCHIVE" name="ROBOTS">
     
  10. xhan

    xhan <span style="color: white; font-weight: 700; backg affiliate

    591
    10
    0
MI