robot.txt

[sesilia]

Robots.txt: This is nothing more than a file, placed in the root of the domain, that is used to inform search bots about the structure of the website. For instance, via the robots.txt file it’s possible to block specific search robots and to restrict the access to specific folders of section inside the website.

 

[ridz ria]

It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want.