Robots.txt?

[jesicawillss]

What is robots.txt? Is it a part of on-page?

 

[arthur]

well, i think so,, it is a part of on page optimization,,but i didnt work on it yet..

 

[remfans]

Robots.txt files are indeed part of on-site optimization. You can use them to give instructions about your site to the web crawlers/spiders. You can ask them not to crawl and/or index at all your entire website, or just simply avoid certain files and/or directories and such.