What Is Robots.txt?

[iwebseo7]

Hello Friends !!
Please tell me abut
What Is Robots.txt?

 

[JavierFlores]

Is a .txt file that mainly serves to indicate to the searchers (Google, Bing...) which folders do not index of your web page, must be hosted in the root of your web and it is not obligatory to have it..

 

[teardroptony9]

Basically, it's a type of instruction/tool web owners use to prevent any robots from visiting a page of a website.

In addition to this:

There are two important considerations when using /robots.txt:

• robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
• the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.

So don't try to use /robots.txt to hide information.

Hope it helps...

 

[amar478]

The robots exclusion protocol (REP), or robots.txt may be a computer file webmasters produce to instruct robots (typically computer programme robots) the way to crawl and index pages on their web site.