Design a web crawler

Requirements

Say these out loud before drawing a single box.

Back-of-the-envelope math

Google-scale-ish. Assumptions labeled.

Architecture

The centerpiece: everything orbits the frontier.

flowchart LR
    SEED[Seed URLs] --> FR[Frontier
per-host queues + priority]
    FR --> SCH[Scheduler
politeness gate]
    SCH --> FE[Fetcher Pool]
    FE --> DNS[(DNS Cache)]
    FE --> RC[(Robots Cache)]
    RC -->|disallow| SKIP[Skip]
    RC -->|allow| GET[HTTP GET]
    GET --> PA[Parser]
    PA --> DED[Dedup
URL Bloom + content simhash]
    DED -->|new| PS[(Page Store)]
    DED -->|duplicate| DROP[Drop]
    PA --> LE[Link Extractor]
    LE --> NORM[URL Normalizer
strip tracking params, fragments]
    NORM --> FR
    PS --> IDX[Indexer
downstream]

The frontier holds one queue per host — this is the key structural decision, because politeness is per-host. The scheduler only hands a fetcher a URL whose host's delay has expired. Fetched pages go through robots check (cached), parse, dedup (URL-level Bloom filter, then content-level simhash for near-duplicates), and link extraction feeds normalized URLs back into the frontier. The indexer consumes the page store downstream and never talks to the crawl loop.

Component deep-dives

Fetching with politeness

sequenceDiagram
    participant S as Scheduler
    participant F as Frontier (per-host queue)
    participant R as Robots Cache
    participant W as Fetcher Worker
    participant H as example.com
    S->>F: next ripe host? (delay expired)
    F-->>S: example.com → /blog/post-9
    S->>R: robots rules for example.com?
    alt cache miss
        S->>H: GET /robots.txt
        H-->>S: rules (cache 24h)
    end
    alt disallowed
        S->>F: drop URL, note the rule
    else allowed
        S->>W: fetch /blog/post-9
        W->>H: GET (1 req — then the host goes quiet)
        H-->>W: 200 HTML
        W->>F: mark host next-allowed = now + delay
    end

Dedup — two gates

sequenceDiagram
    participant P as Parser
    participant B as URL Bloom Filter
    participant C as Content Simhash Index
    participant S as Page Store
    P->>P: normalize URL (lowercase host, strip ?utm_*, #frag)
    P->>B: seen this URL?
    alt yes → drop (URL dupe)
        B-->>P: probably seen — skip fetch entirely
    else no
        P->>P: fetch + parse
        P->>C: simhash(content) near-duplicate?
        alt near-dupe of known page → drop
            C-->>P: Hamming distance < 3 — skip
        else new
            P->>S: store page
            P->>B: add URL
        end
    end

API + data model

A crawler has no public API — its "API" is the frontier's internal contract. This is what the scheduler reads and writes:

frontier_entry { url, host, priority_score, next_fetch_at, retries, discovered_from }
page           { url_hash PK, url, content_hash, fetched_at, content_ptr → blob store }
robots_cache   { host PK, rules, crawl_delay_s, fetched_at }   -- 24h TTL
host_stats     { host PK, urls_seen, urls_fetched, errors_24h, budget_remaining }

Trade-offs

Failure modes

What I'd actually build

Opinionated. Steal this for the interview.

Stack: Go fetcher pool (great at 100K concurrent connections). Frontier as Kafka topics partitioned by host hash + Redis per-host "next allowed at" timestamps (token-bucket politeness). Seen-URL Bloom segments in RocksDB. Page content in S3, metadata in Postgres. Robots cache in Redis, 24h TTL, fail closed. A separate small headless-Chrome fleet for JS-heavy hosts with its own strict budget. Nightly job recomputes priority scores from the link graph.

🔬 Interactive: crawl frontier simulator

A tiny 3-host web. Watch BFS crawl it — with politeness delays, a robots.txt wall, a duplicate URL, and a calendar trap.