Readings:

Web crawling papers archived by ACM:
• Breadth-First Search Crawling Yields High-Quality Pages
• Search Effectiveness with a Breadth-First Crawl
• Crawling a Country: Better Strategies than Breadth-First for Web Page Ordering
• A Memory-Efficient Strategy for Exploring the Web
• A Web Crawler in Perl

Key Concepts:

• Ethical crawling practices
• Obtaining highest-quality pages
• Re-crawling for updates
• Spider traps

Exercises:
Start on the final project.