“mobile”/ Voluum
Welcome to Our Community
Wanting to join the rest of our members? Feel free to sign up today.

Use of robots.txt file?

Discussion in 'Design and Creatives' started by GustavoWoltmann016, Jan 6, 2017.

  1. Marc

    Marc 武士- Spamurai affiliate

    1,512
    1,079
    113
    A robots.txt is a simple texteditor .txt -file which is uploaded into the root directory of your database.
    The file itself only contains simple instructions for the crawlers / bots which pages or parts of the website are not allowed (!!) for them to crawl, and then afterwards not to index in the SERPs (though there is no guarantee that the bots will follow these instructions :().
    Thus a robots.txt is an exclusion-option.

    "User-agent:" is the term for a crawler or bot,
    * stands for "all bots / crawlers",
    and "Disallow:" indicates which sub-pages of the website are not allowed to crawl.

    Example for a simple robots.txt :
    That's it. Then uploaded into the root directory the crawlers should avoid the given sub-pages of the website.


    The XML-Sitemap (= a document in a XML file-format).
    But the job of a sitemap is automatically taken over by a good SEO-plugin,
    such as e.g. Yoast for the WordPress-CMS ;)
     
    Last edited: Apr 30, 2017
    game333 likes this.
  2. CPA Evolution
  3. Alex_Sddd

    Alex_Sddd Affiliate affiliate

    10
    2
    3
    thank you for the hundredth time
     
    DMouse likes this.
  4. Dean Brabin

    Dean Brabin Affiliate affiliate

    32
    2
    8
    Robots.txt file use for website crawling and allow and disallow in search engine.
     
    Last edited by a moderator: May 8, 2017
  5. Ashley Prince

    Ashley Prince Affiliate affiliate

    31
    10
    8
    Hello GustavoWaltmass016,

    The main purpose for robots.txt file is to communicate with the crawlers who would be crawling and indexing your website.

    With the help of robots.txt file, you can tell the crawlers which folder to crawl and which not to crawl.
     
  6. crawfrod

    crawfrod Affiliate affiliate

    3
    0
    1
    Without Robot.txt file your website not appear in Search engine through Robot.txt you get website crawled by SE. Now LongPathTool help to fix in file related errors.
     
  7. MaxAffiliate77

    MaxAffiliate77 Affiliate affiliate

    109
    25
    18
    This file will help you to promote the site in search engines.
     
banners