"Robots.txt" is a approved text file that through its name, has appropriate aim to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can inform robots to not crawl and basis definite files, directories within your site, or at all.