hard google · part of Practice Questions · Senior SWE Roadmap
Requirements to clarify
- Functional: given seed URLs, discover and download web pages, extract links, repeat — store/index the content.
- Non-functional: must be polite (respect robots.txt, rate-limit per domain), must avoid infinite loops/duplicate content, must scale to billions of pages.
Core components
- Frontier (URL queue): a distributed queue of URLs to crawl, typically prioritized and partitioned by domain to enforce per-domain politeness/rate limits without a global bottleneck.
- Fetcher workers: pull a URL, download the page, respecting robots.txt and rate limits for that domain.
- Deduplication: hash page content (not just URL — different URLs can serve identical content) to avoid reprocessing/storing duplicates; a Bloom filter is a common space-efficient way to check “have we seen this URL/hash before.”
- Link extractor: parses downloaded HTML for new links, normalizes them (resolving relative URLs), and feeds them back into the frontier.
- Storage: raw content + extracted metadata, typically in an object store + separate index.
Key tradeoffs
- Politeness (rate-limiting per domain) directly trades off against crawl throughput — partitioning the frontier by domain is what lets you parallelize across domains while still respecting per-domain limits.