What is the main purpose of search engine spiders ?

[Kingson]

What is the main purpose of search engine spiders ?

Ans: Index sites, because even though spiders visit sites, their main function is not only to visit them but to index their contents--- spiders do not monitor sites for unethical activities.


...............................................

|

 

[Jessica123]

RE: What is the  main purpose of search engine spiders ?

Hi,
A crawler or web crawler is a computer program that browses the World Wide Web in a methodical, automated manner or in an orderly fashion, and its function is to perform crawling. While crawling it starts with a list of URLs to visit, called the seeds. As the crawler visits these URLs, it identifies all the hyperlinks in the page and adds them to the list of URLs to visit, called the crawl frontier. URLs from the frontier are recursively visited according to a set of policies.