create Simple robots.txt file

[temi]

I posted this at another forums so I thought someone here may find it useful:


1. If you cut the message below, past it to a text file and call the text file robots.txt you have a robots.txt its that simple.
--------------robots.txt begings-------------------
# robots.txt for www.YourSiteName.com
User-agent: *
Disallow:
----------robots.txt ends---------------------------------------
2. The robots.txt I gave you allows all robots to index all parts of your site, here is what you need to do if you want robots to leave some parts of your site alone.
The 3rd like which says Disallow: just add the folder you want robots to leave a alone, lets say you cgi-bin folder, just add this:
/cgi-bin/
so the new robots.txt that allows robots to every part of your website except the cgi-bin folder would look like this:
# robots.txt for www.YourSiteName.com
User-agent: *
Disallow: /cgi-bin/


Any other folders you want to exclude from robots just add another line with disallow so if you want to exclude robots from another folder called temiPrivateStuff just add this like
Disallow: /temiPrivateStuff/

 

[Bagi Zoltán]

Temi, your method to collect tutorials to improve content is really brilliant. I'm sure I will follow that at my hungarian forum.

 

[temi]

Thanks Bagi, once the tutorials are indexed by search engines and ranked very highly, it will bring additonal traffic to a forum or site so yes, its a good idea to use in your Hungarian forum once its online