“webinar”/ “webinar”/
Welcome to Our Community
Wanting to join the rest of our members? Feel free to sign up today.

Null

 

Whats the best usage of the Robots.txt file ?

Discussion in 'Search Engine Optimisation' started by Bulk Text, Dec 29, 2014.

  1. Bulk Text

    Bulk Text Affiliate affiliate

    1
    0
    0
    What is the robots.txt used for most effectively ? Ive seen people say it can help with SEO and ive also seen others explain how they solve resource abuse with it ?
     
  2. AffBank
  3. saleoid

    saleoid Affiliate affiliate

    48
    0
    0
    Robots.txt file is a text file that helps in restricting web crawler such as Googlebot to crawl certain pages of your website. It uses allow and disallow commands that tell web crawlers which page or URL to crawl and which page not to crawl. The main use of robots.txt file is to restrict those pages that you don't want to index on SERP.
     
  4. richardjack67

    richardjack67 Affiliate affiliate

    10
    0
    0
    If we don't want to index a particular page ..we can restrict it through robot.txt
     
  5. Eve Mistry

    Eve Mistry Affiliate affiliate

    5
    0
    0
    Robot.txt tells search engine that which page ready for crawl or which not. For example if you have sensitive data on your website that you don't want to world see and search engines also do not index this page.
     
  6. jandxb

    jandxb Affiliate affiliate

    32
    4
    0
    robots.txt will be used to block urls which you don't want search engines to visit. Mostly admin pages are blocked in robots.txt
     
  7. kkabir

    kkabir Affiliate affiliate

    6
    0
    0
    basically robots.txt is a way that we can control the restriction of our side.in particular period we can hide our side from viewing
     
  8. robertm96

    robertm96 Affiliate affiliate

    11
    0
    0
    you can control bots access from the robots.txt
     
  9. Manasa

    Manasa Affiliate affiliate

    14
    3
    0
    If there are files and directories you do not want indexed by search engines, you can use a robots.txt file to define where the robots should not go. The robots.txt is a very simple text file placed on your web server.
     
  10. Johncable

    Johncable Affiliate affiliate

    12
    0
    0
    A robots.txt file gives instructions to web robots about the pages the website owner doesn’t wish to be ‘crawled’. For instance, if you didn’t want your images to be listed by Google and other search engines, you’d block them using your robots.txt file.
     
MI