What is robots.txt?

[drarunas]

Robots.txt is a text file used to give instructions to the search engine crawlers about the caching and indexing of a webpage, domain, directory or a file of a website.

 

[tinasmith1097]

Robot.txt is a simple text file through which we can instruct the search engine which page you have to be cache or not.

 

[wicky]

Robot.txt is simple but most important part of any site and without it you could never make your Business familliar worldwide

 

[emaanasad84]

Robots.txt is a file through which you can guide search engines to crawl or not to crawl certain sections of your website. Google specifically follows instructions given in this robots.txt file