What are Spiders, Crawlers and Robots?

[yamamura]

Search engines work by sending out what are called spiders, crawlers, or robots. Essentially, they crawl through cyberspace looking for webpages. When they get to a page, they take a sort of picture of the page, indexing the information on it, and integrating it into their giant database of millions of websites. When you type a word into the search engines looking for websites with content on that topic, in a split second they go through billions of webpages that the spiders and robots have visited. Webpages are typically revisited every few weeks.
..........
|

 

[lillianabe]

Spider, Crawler and Robots all are one type of program which are used by be search engine to update their data directory. All programs work to find out new or latest update and content on web and submit in search engine database.

 

[vgkumar]

Web spiders/robots/crawlers retrieve a web page and then recursively traverse hyperlinks to retrieve further web content. Their accepted behavior is specified by the Robots Exclusion Protocol of the robots.txt file in the web root directory .
................