Use of robots.txt file?

[GustavoWoltmann016]

I wanna know about robots.txt file. What is the use of it?

 

[marketeer]

It tells search engines what to crawl and what to ignore.

 

[ryan412]

^^^ this and pretty important it is if you have a site where you use a CMS or dynamic program languages.

 

[martinsmith]

robots.txt is a text file, web masters create to instruct robots how to crawl and index pages on website....

 

[game333]

It tells robots allow and disallows.

 

[Rosy James]

robots.txt file is used to block website or any specific page to crawl by search engine spider

 

[sambugtreat]

robots.txt file is used block or unblock website pages from search engine bots.

 

[martinsmith]

Robots.txt file is used to block or unblock website or any specific page crawl by search engine bots

 

[iamgaurav]

It basically lets Bots like ( Google,Bing ) to know whether they can crawl your website and what parts of the website should not be crawled.

 

[jJackEvance]

It is for searching robots, like google

 

[floretmediabnglr]

Robot.txt is a simple text file we keep inside our website to tell the search engine's Search bots to what to look and what not to look inside our site.

its not mandatory to use a robot.txt but it is highly recomended by the SEO's and Search engines,to improve the page ranking.
Example:
User-agent: * // * means all the search bots
Disallow: / // / banned to look inside here we can add seperate directory so the search bot wont // check inside that directory

 

[DMouse]

It's worth pointing out that while its supposed to tell bots whether or not they should crawl your site, some of them just ignore it and crawl anyway!

But its good practice to have it and most of the big players do honor it.

 

[riazkhan520]

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. This is really recommended by the SEO's and Search engines,to improve the page ranking.
example: Its looking like
User-agent: Googlebot
Disallow: /images/
Disallow: /temp/
Disallow: /cgi-bin/

 

[Hostingsafety]

Robots.txt file is to let Google know about what pages,files or folders you would like to get indexed.

 

[RH-Calvin]

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.

 

[Alex_Sddd]

this file for a brief description of the site. correct if wrong

 

[DMouse]

this file for a brief description of the site. correct if wrong

No it's not, it's use to tell bots what they can/can't crawl.

Read some of the replies above!

 

[Alex_Sddd]

No it's not, it's use to tell bots what they can/can't crawl.

Read some of the replies above!

ooops, i`m sorry. and which contains a description of the site?

 

[Alex_Sddd]

No it's not, it's use to tell bots what they can/can't crawl.

Read some of the replies above!

i mean this

 

[DMouse]

ooops, i`m sorry. and which contains a description of the site?

Look up meta tags, can't look up properly ATM as on phone on poor connection!!