Development and Evolution of Web Crawlers:Current Status and Future Perspectives

Internet provides access to a huge repository of data making the search and retrieval of the required information trivial. Dynamic increase in the complexity and the volume of the information further makes the effective search necessary and also challenging. Web crawling is the mechanism used by the various search engines to collect the pages from the World Wide Web. The crawlers need to be intelligent and adaptive with respect to the environment they are acting on. With the existence of different strategies that rank the pages, still getting the required precision and recall values is a challenge. Other factors that add up the complexity are the memory constraints and the hit-rate that takes place. Considering the recent trends and evolutions, developing efficient web crawlers is still an open issue attracting many researchers. This article presents a survey of origin, need and the current advances in the evolution of web crawlers.