Web Crawling

Front Cover
Now Publishers Inc, 2010 - Computers - 86 pages
This is a survey of the science and practice of web crawling. While at first glance web crawling may appear to be merely an application of breadth-first-search, the truth is that there are many challenges ranging from systems concerns such as managing very large data structures, to theoretical questions such as how often to revisit evolving content sources. This survey outlines the fundamental challenges and describes the state-of-the-art models and solutions. It also highlights avenues for future work.
 

What people are saying - Write a review

We haven't found any reviews in the usual places.

Contents

Introduction
1
Crawl Ordering Problem
19
Incremental Crawl Ordering
41
Copyright

Common terms and phrases

Bibliographic information