“webinar”/ “VOLUUM”/
Welcome to Our Community
Wanting to join the rest of our members? Feel free to sign up today.

Use of robots.txt file?

Discussion in 'Design and Creatives' started by GustavoWoltmann016, Jan 6, 2017.

  1. GustavoWoltmann016

    GustavoWoltmann016 Affiliate affiliate

    31
    12
    8
    I wanna know about robots.txt file. What is the use of it?
     
  2. Voluum
  3. marketeer

    marketeer Affiliate affiliate

    62
    34
    18
    It tells search engines what to crawl and what to ignore.
     
    Albertomax1, game333 and internalsoul like this.
  4. ryan412

    ryan412 Affiliate affiliate

    23
    3
    3
    ^^^ this and pretty important it is if you have a site where you use a CMS or dynamic program languages.
     
  5. martinsmith

    martinsmith Affiliate affiliate

    159
    24
    18
    robots.txt is a text file, web masters create to instruct robots how to crawl and index pages on website....
     
  6. game333

    game333 Moderator moderator affiliate

    1,446
    1,395
    113
    It tells robots allow and disallows.
     
    Marc likes this.
  7. Rosy James

    Rosy James Affiliate affiliate

    12
    1
    1
    robots.txt file is used to block website or any specific page to crawl by search engine spider
     
    Marc likes this.
  8. sambugtreat

    sambugtreat Affiliate affiliate

    9
    0
    1
    robots.txt file is used block or unblock website pages from search engine bots.
     
  9. martinsmith

    martinsmith Affiliate affiliate

    159
    24
    18
    Robots.txt file is used to block or unblock website or any specific page crawl by search engine bots
     
  10. iamgaurav

    iamgaurav Affiliate Manager Affiliate Manager affiliate

    13
    6
    3
    It basically lets Bots like ( Google,Bing ) to know whether they can crawl your website and what parts of the website should not be crawled.
     
  11. jJackEvance

    jJackEvance Banned affiliate

    23
    4
    8
    It is for searching robots, like google
     
  12. floretmediabnglr

    floretmediabnglr Affiliate affiliate

    4
    0
    1
    Robot.txt is a simple text file we keep inside our website to tell the search engine's Search bots to what to look and what not to look inside our site.

    its not mandatory to use a robot.txt but it is highly recomended by the SEO's and Search engines,to improve the page ranking.
    Example:
    User-agent: * // * means all the search bots
    Disallow: / // / banned to look inside here we can add seperate directory so the search bot wont // check inside that directory
     
  13. DMouse

    DMouse Affiliate affiliate

    76
    55
    18
    It's worth pointing out that while its supposed to tell bots whether or not they should crawl your site, some of them just ignore it and crawl anyway!

    But its good practice to have it and most of the big players do honor it.
     
    Marc likes this.
  14. riazkhan520

    riazkhan520 Affiliate affiliate

    19
    4
    3
    The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. This is really recommended by the SEO's and Search engines,to improve the page ranking.
    example: Its looking like
    User-agent: Googlebot
    Disallow: /images/
    Disallow: /temp/
    Disallow: /cgi-bin/
     
  15. Hostingsafety

    Hostingsafety Affiliate affiliate

    70
    5
    8
    Robots.txt file is to let Google know about what pages,files or folders you would like to get indexed.
     
  16. RH-Calvin

    RH-Calvin Affiliate affiliate

    176
    31
    28
    Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
     
  17. Alex_Sddd

    Alex_Sddd Affiliate affiliate

    10
    2
    3
    this file for a brief description of the site. correct if wrong
     
  18. DMouse

    DMouse Affiliate affiliate

    76
    55
    18
    No it's not, it's use to tell bots what they can/can't crawl.

    Read some of the replies above!
     
    Alex_Sddd likes this.
  19. Alex_Sddd

    Alex_Sddd Affiliate affiliate

    10
    2
    3
    ooops, i`m sorry. and which contains a description of the site?
     
  20. Alex_Sddd

    Alex_Sddd Affiliate affiliate

    10
    2
    3
    i mean this
     

    Attached Files:

  21. DMouse

    DMouse Affiliate affiliate

    76
    55
    18
    Look up meta tags, can't look up properly ATM as on phone on poor connection!!
     
    Alex_Sddd likes this.
MI