Use of robots.txt file?

[Marc]

I wanna know about robots.txt file. What is the use of it?

A robots.txt is a simple texteditor .txt -file which is uploaded into the root directory of your database.
The file itself only contains simple instructions for the crawlers / bots which pages or parts of the website are not allowed (!!) for them to crawl, and then afterwards not to index in the SERPs (though there is no guarantee that the bots will follow these instructions ).
Thus a robots.txt is an exclusion-option.

"User-agent:" is the term for a crawler or bot,
* stands for "all bots / crawlers",
and "Disallow:" indicates which sub-pages of the website are not allowed to crawl.

Example for a simple robots.txt :

User-agent: *
Disallow: /about us
Disallow: /legal notes
Disallow: /private policies

That's it. Then uploaded into the root directory the crawlers should avoid the given sub-pages of the website.

which contains a description of the site?

The XML-Sitemap (= a document in a XML file-format).
But the job of a sitemap is automatically taken over by a good SEO-plugin,
such as e.g. Yoast for the WordPress-CMS

 

[Alex_Sddd]

Look up meta tags, can't look up properly ATM as on phone on poor connection!!

thank you for the hundredth time

 

[Dean Brabin]

Robots.txt file use for website crawling and allow and disallow in search engine.

 

[Ashley Prince]

I wanna know about robots.txt file. What is the use of it?

Hello GustavoWaltmass016,

The main purpose for robots.txt file is to communicate with the crawlers who would be crawling and indexing your website.

With the help of robots.txt file, you can tell the crawlers which folder to crawl and which not to crawl.

 

[crawfrod]

Without Robot.txt file your website not appear in Search engine through Robot.txt you get website crawled by SE. Now LongPathTool help to fix in file related errors.

 

[MaxAffiliate77]

This file will help you to promote the site in search engines.