Robots.txt files are indeed part of on-site optimization. You can use them to give instructions about your site to the web crawlers/spiders. You can ask them not to crawl and/or index at all your entire website, or just simply avoid certain files and/or directories and such.
Welcome to Our Community
Wanting to join the rest of our members? Feel free to sign up today.
This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.
Attention: "role based name" email prefixes such as support, webmaster, info, contact, forum, service, affiliate, admin, traffic, partners, marketing, etc., for our member accounts are not permitted. They are acceptable for a vendor listing, but not for the member email. This is due to bulk email processors scrubbing them for delivery due to extremely high bounce rates. Please apply for membership using a personalized email prefix. Something like bob@xxxxx.com.
If we discover you signed up with a role based name we will give you only one message to change it. After 3 days your account will be removed if the email is not changed.