Robot.txt is a simple text file we keep inside our website to tell the search engine's Search bots to what to look and what not to look inside our site.
its not mandatory to use a robot.txt but it is highly recomended by the SEO's and Search engines,to improve the page ranking.
Example...
This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.
Attention: "role based name" email prefixes such as support, webmaster, info, contact, forum, service, affiliate, admin, traffic, partners, marketing, etc., for our member accounts are not permitted. They are acceptable for a vendor listing, but not for the member email. This is due to bulk email processors scrubbing them for delivery due to extremely high bounce rates. Please apply for membership using a personalized email prefix. Something like bob@xxxxx.com.
If we discover you signed up with a role based name we will give you only one message to change it. After 3 days your account will be removed if the email is not changed.