what is the Robots.txt?

[Danielnash]

Hi everybody,


tell me about Robots.txt file?


Thanks

 

[justinmarch]

See this The Web Robots Pages

 

[Alfredwinston]

Google it dude and you can find lots of info regarding it. Why making a thread for these simple things. Anyhow let me say, Robots.txr is the instructor for search engine bots that crawl ur site. It says which page to be indexed, which to be skipped etc..,

 

[airforce1]

Simply speaking, robots.txt is a file under your site root for guiding search engine crawlers how to crawl your sites. But I think it might also a security risk for misusing this file.

 

[extomas]

The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code.

 

[shannonbburns]

If you want to hide your site from search engine or you want to hide particular directory or file of your site from search engines crawler or want to give permission to particular search engine for indexing or link follow... then you need to use Robots.txt file.

 

[ranacseruet]

it is used to restrict search engine robots what to crawl and what not to... such as you might not want search engine to crawl your admin panel page's secret information.....

 

[temi]

Hi everybody,


tell me about Robots.txt file?


Thanks

You can find a live example of robots .txt file here

 

[rohnsmith]

If you do not want google robot or any other search engine robot to index your page then you can use this file and you need to upload it in root directory .There are many tools like google web master tool which you can use to generate robot .txt

 

[pow-wow]

A robots.txt is a file placed on your server to tell the various search engine spiders not to crawl or index certain sections or pages of your site.