Robots.txt

[vjackcon]

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter.
Structure of a Robots.txt File

The structure of a robots.txt is pretty simple (and barely flexible) – it is an endless list of user agents and disallowed files and directories. Basically, the syntax is as follows:

User-agent:

Disallow:

“User-agent” are search engines' crawlers and disallow: lists the files and directories to be excluded from indexing. In addition to “user-agent:” and “disallow:” entries, you can include comment lines – just put the # sign at the beginning of the line:

# All user agents are disallowed to see the /temp directory.

User-agent: *

Disallow: /temp/

 

[DarrelCornell]

"Robots.txt" is a approved text file that through its name, has appropriate aim to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can inform robots to not crawl and basis definite files, directories within your site, or at all.

 

[markhardy20]

Robots.txt is a regular text file here. Its name has a special meaning to the majority of honorable robots on the web. This file is very helpful to us.

 

[briensmith]

It contains restrictions for Web Spiders, telling them where they have permission to search. It is like defining rules for search engine spiders (robots) what to follow and what not to. It provides you with more functionality than Meta robots tag which is available only partially to control behaviour of search engines. You can use it to prevent indexing totally, prevent certain areas of your site from being indexed or to issue individual indexing instructions to specific search engines. Robot.txt protocols are simply advisory though. There is no law requiring websites to have Robot.txt files, or to use them on their web pages.



Thanks,

 

[ultimatewebby]

from what i have learned.. .robots are there to be used as parameters.. .or in a way limit search engines for your data base to be indexed.. .so, i guess its kinda in your side.. .cause it sounds so cool.. .

 

[ngseo]

Robot.txt is a text file where in search engine spiders will read important pages from your sites and disregard unnecessary pages.