# Design a Web Crawler | System Design Question

A web crawler is the program behind every search engine: it starts from a handful of URLs, downloads those pages, pulls the links out of them, and follows those links to find more pages, over and over, until it has walked a large chunk of the web. Googlebot is one. So is the bot that fills the Internet Archive. In an interview, "design a web crawler" looks deceptively small, three lines of pseudo-code, and then explodes the moment you ask "at what scale?". We will build it the way a real interview runs: the questions first, then a design reasoned into existence one number at a time, then the deep dives where this problem actually lives.

If you have not seen the four-step framework this walkthrough follows, it is worth reading the [60-minute system design interview framework](/60-mins-system-design-interview-framework) first; here we put it to work.

## The three-line algorithm, and why it is a trap

Strip a crawler down and it is genuinely this simple:

```typescript
// The whole idea, minus every hard part.
async function crawl(seeds: string[]): Promise<void> {
  const frontier: string[] = [...seeds];   // URLs still to visit
  const seen = new Set<string>(seeds);     // URLs we have already queued

  while (frontier.length > 0) {
    const url = frontier.shift()!;         // take the next URL
    const html = await download(url);      // fetch the page
    for (const link of extractLinks(html)) {
      if (!seen.has(link)) {
        seen.add(link);
        frontier.push(link);               // discover new work
      }
    }
  }
}
```

That runs fine against one small website. Point it at *the web* and every line breaks: `frontier` becomes hundreds of millions of URLs that do not fit in memory, `seen` becomes a set of billions of strings, `download` hammers one server so hard it looks like an attack, and `while (frontier.length > 0)` never ends because the web is effectively infinite and full of traps designed to loop you forever.

Everything interesting about this problem is what you add to keep that loop alive at scale. So let's establish the scale.

## Step 1: Ask questions, and pin down the scale

"Design a web crawler" is under-specified on purpose. A crawler for one company's docs site and a crawler that feeds a global search engine share three lines of code and nothing else. Your first job is to find out which one you are building, and to keep the conversation moving even when the answers are messy. A realistic exchange:

> **You:** What is the crawler *for*? Search-engine indexing, archiving, price monitoring, training data?
>
> **Interviewer:** Search-engine indexing. Assume we are feeding a search index.
>
> **You:** Roughly how many pages per month should it pull down?
>
> **Interviewer:** Call it one billion new pages a month.
>
> **You:** What content types? HTML only, or also images, PDFs, video?
>
> **Interviewer:** HTML only for now.

Good, clean answers so far. Now the interviewer starts to push back, and this is the part the polished write-ups skip:

> **You:** Do we store the raw pages we download, and if so for how long?
>
> **Interviewer:** You tell me.

That "you tell me" is where people freeze. Don't. **Propose a number, say your reasoning out loud, and write it down as an assumption:** "For a search index you re-process pages over time, so I'll assume we keep raw HTML for five years and design the storage for that, stop me if that is way off." Now *you* own that decision instead of stalling on it.

> **You:** Should we re-crawl pages to catch edits, or is one visit per URL enough?
>
> **Interviewer:** Yes, pages change, we need to keep the index fresh.
>
> **Interviewer:** Oh, and while you're at it, can it also detect copyright violations across the sites we crawl?

There is the curveball. A less experienced interviewer will happily bolt on scope mid-problem. Don't design it, and don't ignore it either, **name its one-line cost and park it:** "That is a separate analysis module that consumes crawled pages, I'll make sure the design is extensible enough to plug it in, but I'll get the core crawler working first." Then move on.

The rule underneath all of it: **when the interviewer gives you an answer, use it; when they punt, assume and state it; when they pile on scope, name its cost and keep going.** Never wait for a tidy answer that may not come.

With that, write the scope as precise points, not a paragraph.

### Functional requirements

- Start from a set of seed URLs and download the pages they point to.
- Extract the links (URLs) from every downloaded page and follow them.
- Store the raw HTML of crawled pages (assumption: retained for five years).
- Ignore duplicate content, the same page reached through a different URL.
- Re-crawl pages periodically so the data set stays fresh.
- HTML only; other content types are out of scope but the design should extend to them.

### Non-functional requirements

- **Scalability.** The web is billions of pages; the crawl must run massively in parallel.
- **Politeness.** Never flood one website with requests, that is rude and looks like a denial-of-service attack.
- **Robustness.** The web is hostile, bad HTML, dead servers, redirect loops, malicious links, must not crash the crawler.
- **Freshness.** Important, fast-changing pages get re-crawled more often than static ones.
- **Extensibility.** Adding a new content type (images, PDFs) should be a plug-in, not a rewrite.

### A back-of-the-envelope estimate

Turn the one billion pages a month into numbers. (For the mental-math shortcuts here, see [easier back-of-the-envelope estimation](/easier-back-of-the-envelope-estimation).)

```text
Pages per month      = 1,000,000,000
Seconds per month    ~ 30 x 24 x 3600 = 2,592,000  (~2.5 million)
Average download QPS = 1,000,000,000 / 2,592,000  ~ 400 pages/second
Peak QPS             = 2 x average                = ~800 pages/second

Average page size    ~ 500 KB
Storage per month    = 1,000,000,000 x 500 KB     = 500 TB / month
Storage for 5 years  = 500 TB x 12 x 5            = 30 PB
```

Two facts fall out and shape everything downstream. First, ~400 pages/second sustained (800 at peak) is not something one machine does politely, one host at a time, so the crawler must be **distributed across many workers**. Second, **30 petabytes** of retained HTML is far too big for memory or a single disk, so storage must be **sharded across many machines**, with only hot pages cached in memory. Keep both numbers in your pocket; we will reason each component out of them.

## Step 2: Build the high-level design from the numbers

I'll grow the architecture component by component, each one justified by a number or a requirement, rather than dropping a finished diagram on the table.

**Where does work live? The URL Frontier.** The naive loop kept a `frontier` array in memory. At web scale that queue holds hundreds of millions of URLs, so it cannot be a language array, and, as we will see, it is not even a plain queue. I'll call it the **URL Frontier**: the component that stores all URLs discovered but not yet downloaded. Think of it as the crawler's to-do list. It is a FIFO (first-in-first-out) queue at heart, first URL in is the first one out, but it grows two extra jobs later (politeness and priority) that make it the most interesting box in the whole system. For now: seeds go in, workers pull URLs out.

**Who fetches pages? The HTML Downloader.** At ~800 pages/second peak, one thread blocking on one slow server would tank throughput, so the downloader is **many worker threads across many machines**, each pulling URLs from the frontier and fetching over HTTP. This is the component the whole "distributed" requirement lives in.

**How does a URL become a fetch? The DNS Resolver.** A URL names a host (`en.wikipedia.org`); to open a connection the crawler needs its IP address, and DNS (the Domain Name System, the internet's phone book that turns hostnames into IP addresses) is what translates it. A single DNS lookup can take 10 to 200 ms, and at hundreds of lookups per second that becomes the bottleneck, so the downloader talks to a **DNS resolver with a local cache** rather than asking the network every time. More on why this matters so much in the deep dive.

**What cleans up the raw bytes? The Content Parser.** Downloaded HTML is frequently malformed, truncated, or gigantic. Parsing and validating it is CPU-heavy, and if the downloader threads did it inline, fetching would stall. So parsing is its **own component**, downloaders hand raw pages off and go back to fetching.

**How do we skip duplicate pages? "Content Seen?".** Studies of the web find that roughly 29% of pages are duplicate content, the same article mirrored on many URLs. Storing all of it wastes a big fraction of our 30 PB. So before storing, a **"Content Seen?"** check asks whether we already have this exact page. Comparing full HTML character by character across billions of pages is hopeless; instead we hash each page (turn its content into a short fixed-length fingerprint) and compare fingerprints. Same fingerprint, drop the page.

**Where do 30 PB of pages go? Content Storage.** Too big for memory, too big for one disk. So content lives in a **distributed store, mostly on disk, with popular pages cached in memory** to cut latency.

**How do we find more work? URL Extractor, then URL Filter.** The parser's clean HTML goes to a **URL Extractor** that pulls out every link and expands relative links (`/wiki/Foo`) into absolute ones (`https://en.wikipedia.org/wiki/Foo`). Those pass through a **URL Filter** that throws away file types we do not want, known-bad or blocklisted sites, and malformed URLs.

**How do we avoid crawling the same URL twice? "URL Seen?".** The naive loop used `seen = new Set()`. At web scale that set holds *billions* of URLs and cannot fit in memory as plain strings. The **"URL Seen?"** component answers "have we already downloaded or queued this URL?" using a **bloom filter** (a compact probabilistic structure that answers set-membership in a tiny fraction of the memory a real set needs) backed by a hash table. We won't re-derive how a bloom filter works, it is covered in depth in [what is a bloom filter](/what-is-a-bloom-filter); here it is exactly the right tool, because it lets us reject already-seen URLs using a few bits each instead of storing the whole string. New URLs go back into the frontier; the loop closes.

Chain those together and the architecture is the culmination, not the starting point:

![High-level architecture of a web crawler: seed URLs feed the URL Frontier, the HTML Downloader (backed by a DNS cache and robots.txt cache) fetches pages, the Content Parser validates them, a Content Seen check dedupes before Content Storage, and a URL Extractor to URL Filter to URL Seen path loops new URLs back into the frontier](https://mermaid.ink/img/Zmxvd2NoYXJ0IFRECiAgU2VlZChbU2VlZCBVUkxzXSkgLS0+IEZyb250aWVyW1VSTCBGcm9udGllcjxici8+cXVldWUgb2YgVVJMcyBzdGlsbCB0byBjcmF3bF0KICBGcm9udGllciAtLT4gRG93bmxvYWRlcltIVE1MIERvd25sb2FkZXI8YnIvPm11bHRpLXRocmVhZGVkIGZldGNoXQogIEROU1tETlMgUmVzb2x2ZXI8YnIvPndpdGggbG9jYWwgY2FjaGVdIC0uLT4gRG93bmxvYWRlcgogIFJvYm90c1tyb2JvdHMudHh0IGNhY2hlXSAtLi0+IERvd25sb2FkZXIKICBEb3dubG9hZGVyIC0tPiBQYXJzZXJbQ29udGVudCBQYXJzZXI8YnIvPnBhcnNlIGFuZCB2YWxpZGF0ZSBIVE1MXQogIFBhcnNlciAtLT4gQ1NlZW57Q29udGVudCBTZWVufQogIENTZWVuIC0tIGR1cGxpY2F0ZSAtLT4gRHJvcDFbRGlzY2FyZCBwYWdlXQogIENTZWVuIC0tIG5ldyAtLT4gU3RvcmFnZVsoQ29udGVudCBTdG9yYWdlPGJyLz5kaXNrIHBsdXMgaG90IGNhY2hlKV0KICBDU2VlbiAtLSBuZXcgLS0+IEV4dHJhY3RvcltVUkwgRXh0cmFjdG9yPGJyLz5wdWxsIG91dCBsaW5rc10KICBFeHRyYWN0b3IgLS0+IEZpbHRlcltVUkwgRmlsdGVyPGJyLz5kcm9wIGJhZCBleHRlbnNpb25zIGFuZCBibG9ja2xpc3RdCiAgRmlsdGVyIC0tPiBVU2VlbntVUkwgU2Vlbn0KICBVU2VlbiAtLSBhbHJlYWR5IHNlZW4gLS0+IERyb3AyW0Rpc2NhcmQgVVJMXQogIFVTZWVuIC0tIG5ldyAtLT4gRnJvbnRpZXIKICBjbGFzc0RlZiBzZWVkIGZpbGw6IzI1NjNlYixzdHJva2U6IzFlNDBhZixjb2xvcjojZmZmZmZmOwogIGNsYXNzRGVmIGJveCBmaWxsOiNlZmY2ZmYsc3Ryb2tlOiMyNTYzZWIsY29sb3I6IzFlM2E4YTsKICBjbGFzc0RlZiBzdG9yZSBmaWxsOiNkYmVhZmUsc3Ryb2tlOiMxZTQwYWYsY29sb3I6IzFlM2E4YTsKICBjbGFzc0RlZiBkcm9wIGZpbGw6I2ZlZTJlMixzdHJva2U6I2I5MWMxYyxjb2xvcjojN2YxZDFkOwogIGNsYXNzRGVmIGRlYyBmaWxsOiNmZWY5YzMsc3Ryb2tlOiNjYThhMDQsY29sb3I6IzcxM2YxMjsKICBjbGFzcyBTZWVkIHNlZWQ7CiAgY2xhc3MgRnJvbnRpZXIsRG93bmxvYWRlcixETlMsUm9ib3RzLFBhcnNlcixFeHRyYWN0b3IsRmlsdGVyIGJveDsKICBjbGFzcyBTdG9yYWdlIHN0b3JlOwogIGNsYXNzIERyb3AxLERyb3AyIGRyb3A7CiAgY2xhc3MgQ1NlZW4sVVNlZW4gZGVjOwo=?type=png&bgColor=FFFFFF)

Read the loop once end to end: seed URLs enter the frontier; downloaders fetch (resolving DNS and obeying robots.txt); the parser cleans the HTML; "Content Seen?" drops duplicates and stores the rest; the extractor and filter produce candidate URLs; "URL Seen?" drops ones we have already queued and feeds genuinely new ones back into the frontier. That cycle *is* the crawler.

## Step 3: The deep dives, where this problem actually lives

The high-level boxes are the easy 40%. The interview lives in the next four questions.

### Deep dive 1: Which order do we crawl in? BFS, not DFS

Picture the web as a directed graph, pages are nodes, links are the arrows between them:

![The web drawn as a directed graph: a seed URL links to pages A and B, which link onward to C, D, E and F, with page C linking back to A to form a cycle](https://mermaid.ink/img/Z3JhcGggTFIKICBTKFtTZWVkIFVSTF0pIC0tPiBBW1BhZ2UgQV0KICBTIC0tPiBCW1BhZ2UgQl0KICBBIC0tPiBDW1BhZ2UgQ10KICBBIC0tPiBEW1BhZ2UgRF0KICBCIC0tPiBECiAgQiAtLT4gRVtQYWdlIEVdCiAgQyAtLT4gQQogIEQgLS0+IEZbUGFnZSBGXQogIEUgLS0+IEYKICBjbGFzc0RlZiBzZWVkIGZpbGw6IzI1NjNlYixzdHJva2U6IzFlNDBhZixjb2xvcjojZmZmZmZmOwogIGNsYXNzRGVmIHBhZ2UgZmlsbDojZWZmNmZmLHN0cm9rZTojMjU2M2ViLGNvbG9yOiMxZTNhOGE7CiAgY2xhc3MgUyBzZWVkOwogIGNsYXNzIEEsQixDLEQsRSxGIHBhZ2U7Cg==?type=png&bgColor=FFFFFF)

Notice the arrow from C back to A, that cycle is why "URL Seen?" is not optional; without it the crawler loops on A, C, A, C forever.

Traversing a graph, you pick DFS or BFS.

- **DFS (depth-first search)** dives as deep as it can down one path before backing up. On the web, "as deep as it can" can be effectively unbounded, one site can generate infinitely deep URL paths, so DFS can wander down a single site forever and never come back. Bad fit.
- **BFS (breadth-first search)** visits everything one hop from the seeds, then everything two hops out, and so on, level by level. It is a plain FIFO queue: dequeue a URL, enqueue its children at the back. This is why the frontier is a queue.

So crawlers use BFS. But plain BFS has two problems that force the frontier to grow up:

1. **It is impolite.** Most links on a page point back to the *same host*, look at any Wikipedia article, nearly every link is another `wikipedia.org` page. So a FIFO frontier fills up with URLs from one host, and parallel downloaders all hit that one host at once. That is a flood.
2. **It ignores importance.** BFS treats a random forum post and a site's homepage as equal. But the homepage is worth crawling first and more often. Plain FIFO has no notion of priority.

Fixing these two is the entire point of the next deep dive.

### Deep dive 2: The URL Frontier, done properly

The frontier is where politeness and priority get engineered in. The trick is two layers of queues:

![The URL Frontier internals: a Prioritizer scores incoming URLs into front queues f1 to fn by priority; a front-queue selector biased toward high priority feeds a back-queue router; a mapping table sends each host to its own back queue b1 to bn; a back-queue selector assigns each back queue to one worker thread](https://mermaid.ink/img/Zmxvd2NoYXJ0IFRECiAgSW4oW05ldyBVUkxzIGluXSkgLS0+IFByaW9bUHJpb3JpdGl6ZXI8YnIvPnNjb3JlIGJ5IFBhZ2VSYW5rLCB0cmFmZmljLCBmcmVzaG5lc3NdCiAgUHJpbyAtLT4gRjFbRnJvbnQgcXVldWUgZjE8YnIvPmhpZ2ggcHJpb3JpdHldCiAgUHJpbyAtLT4gRjJbRnJvbnQgcXVldWUgZjI8YnIvPm1lZGl1bV0KICBQcmlvIC0tPiBGbltGcm9udCBxdWV1ZSBmbjxici8+bG93IHByaW9yaXR5XQogIEYxIC0tPiBGU2VsW0Zyb250IHF1ZXVlIHNlbGVjdG9yPGJyLz5iaWFzZWQgcGljayB0b3dhcmQgaGlnaCBwcmlvcml0eV0KICBGMiAtLT4gRlNlbAogIEZuIC0tPiBGU2VsCiAgRlNlbCAtLT4gUm91dGVyW0JhY2sgcXVldWUgcm91dGVyXQogIFJvdXRlciAtLT4gTWFwW01hcHBpbmcgdGFibGU8YnIvPmhvc3QgbWFwcyB0byBvbmUgYmFjayBxdWV1ZV0KICBNYXAgLS0+IEIxW0JhY2sgcXVldWUgYjE8YnIvPm9ubHkgaG9zdCBBXQogIE1hcCAtLT4gQjJbQmFjayBxdWV1ZSBiMjxici8+b25seSBob3N0IEJdCiAgTWFwIC0tPiBCbltCYWNrIHF1ZXVlIGJuPGJyLz5vbmx5IGhvc3QgTl0KICBCMSAtLT4gQlNlbFtCYWNrIHF1ZXVlIHNlbGVjdG9yXQogIEIyIC0tPiBCU2VsCiAgQm4gLS0+IEJTZWwKICBCU2VsIC0tPiBXMVtXb3JrZXIgdGhyZWFkIDFdCiAgQlNlbCAtLT4gVzJbV29ya2VyIHRocmVhZCAyXQogIEJTZWwgLS0+IFduW1dvcmtlciB0aHJlYWQgTl0KICBzdWJncmFwaCBGcm9udEhhbGZbRnJvbnQgcXVldWVzIG1hbmFnZSBQUklPUklUWV0KICAgIEYxCiAgICBGMgogICAgRm4KICBlbmQKICBzdWJncmFwaCBCYWNrSGFsZltCYWNrIHF1ZXVlcyBtYW5hZ2UgUE9MSVRFTkVTU10KICAgIEIxCiAgICBCMgogICAgQm4KICBlbmQKICBjbGFzc0RlZiBzZWVkIGZpbGw6IzI1NjNlYixzdHJva2U6IzFlNDBhZixjb2xvcjojZmZmZmZmOwogIGNsYXNzRGVmIGJveCBmaWxsOiNlZmY2ZmYsc3Ryb2tlOiMyNTYzZWIsY29sb3I6IzFlM2E4YTsKICBjbGFzc0RlZiBxIGZpbGw6I2RiZWFmZSxzdHJva2U6IzFlNDBhZixjb2xvcjojMWUzYThhOwogIGNsYXNzRGVmIHdvcmtlciBmaWxsOiNkY2ZjZTcsc3Ryb2tlOiMxNTgwM2QsY29sb3I6IzE0NTMyZDsKICBjbGFzcyBJbiBzZWVkOwogIGNsYXNzIFByaW8sRlNlbCxSb3V0ZXIsTWFwLEJTZWwgYm94OwogIGNsYXNzIEYxLEYyLEZuLEIxLEIyLEJuIHE7CiAgY2xhc3MgVzEsVzIsV24gd29ya2VyOwo=?type=png&bgColor=FFFFFF)

**Front queues handle priority.** A **Prioritizer** scores each incoming URL, by PageRank (Google's original measure of a page's importance, derived from how many other pages link to it), site traffic, update frequency, whatever signals you have, and drops it into one of several front queues f1 (high) to fn (low). A **front-queue selector** then picks the next URL with a bias toward the high-priority queues: it usually pulls from f1, sometimes from f2, rarely from fn. Important pages get crawled sooner, without starving the rest.

**Back queues handle politeness.** The selected URL goes to a **back-queue router** that consults a **mapping table** from host to queue, and drops the URL into that host's dedicated back queue. The invariant is the key: **each back queue holds URLs from exactly one host.** A **back-queue selector** assigns each back queue to a single **worker thread**, so one host is only ever downloaded by one worker, one page at a time, with a delay between fetches. That is politeness enforced structurally, not by hope.

In code, the politeness half is a per-host queue with a delay:

```typescript
type Host = string;

// One worker owns one host's queue and never fetches it faster than the delay.
class PoliteHostQueue {
  private queue: string[] = [];
  private lastFetch = 0;
  constructor(private readonly host: Host, private readonly delayMs = 1000) {}

  enqueue(url: string): void {
    this.queue.push(url);
  }

  async runOnce(download: (url: string) => Promise<string>): Promise<void> {
    const url = this.queue.shift();
    if (!url) return;

    const waitMs = this.lastFetch + this.delayMs - Date.now();
    if (waitMs > 0) {
      await new Promise((r) => setTimeout(r, waitMs)); // stay polite
    }

    await download(url);
    this.lastFetch = Date.now();
  }
}
```

One knob to call out: a fixed `delayMs` is the simple version. A better crawler sets the delay per host, faster for a big site that can take it, slower for a small server, and honors any `Crawl-delay` the site declares in its robots.txt (next section).

**Freshness** rides on the same priority machinery. The web changes constantly, so downloaded pages have to be re-crawled, but re-crawling all one billion every cycle is wasteful. Instead, re-crawl by importance and by observed change rate: a news homepage that updates hourly gets re-queued far more often than an archived page that has not changed in a year. In practice you feed re-crawl URLs back through the same Prioritizer, so freshness is just another input to priority.

**Where does the frontier physically live?** At web scale it holds hundreds of millions of URLs, too many for memory. All in memory is not durable (a crash loses the whole to-do list); all on disk is too slow to be the hot path. So use a **hybrid**: the bulk of the queues live on disk for durability and size, with in-memory buffers for the enqueue and dequeue ends, flushed to disk periodically. Fast where it is hot, durable and roomy where it is cold.

### Deep dive 3: The HTML Downloader, robots.txt and the DNS wall

**robots.txt comes first.** Before touching a site, a well-behaved crawler fetches `https://example.com/robots.txt`, the Robots Exclusion Protocol, a plain-text file where a site states which paths crawlers may and may not fetch. Ignore it and you get blocked, or sued. A snippet from Amazon's:

```text
User-agent: Googlebot
Disallow: /creatorhub/*
Disallow: /gp/aw/cr/
```

You do not re-download robots.txt on every request, that would be its own flood, you **cache it per host** and refresh periodically. That is the "robots.txt cache" box hanging off the downloader in the architecture diagram.

```typescript
// Cache robots rules per host so we fetch the file once, not per URL.
class RobotsCache {
  private cache = new Map<Host, { disallow: string[]; fetchedAt: number }>();
  private ttlMs = 24 * 60 * 60 * 1000; // refresh daily

  async isAllowed(host: Host, path: string): Promise<boolean> {
    let rules = this.cache.get(host);
    if (!rules || Date.now() - rules.fetchedAt > this.ttlMs) {
      rules = await this.fetchAndParse(host);
      this.cache.set(host, rules);
    }
    return !rules.disallow.some((prefix) => path.startsWith(prefix));
  }

  private async fetchAndParse(host: Host): Promise<{ disallow: string[]; fetchedAt: number }> {
    // fetch https://<host>/robots.txt, parse the Disallow lines for our user-agent
    return { disallow: [], fetchedAt: Date.now() };
  }
}
```

**DNS is the sneaky bottleneck.** Every fetch needs the host's IP, and a DNS lookup takes 10 to 200 ms. Many DNS client libraries are *synchronous*, one lookup blocks the calling thread until it returns, so at hundreds of fetches per second, DNS resolution, not the actual page download, becomes what limits you. The fix is the "DNS Resolver with cache" box: keep a local hostname to IP cache (hosts repeat constantly, most links stay on the same host), refreshed by a background job. A cache hit turns a 200 ms wait into microseconds.

Beyond those two, three more optimizations:

- **Distributed crawl.** Split the URL space across many servers, each responsible for a slice, each running many threads. This is how you reach 800 pages/second at all.
- **Geographic locality.** Put crawl servers near the sites they fetch, a server in Europe crawling European sites has a shorter round-trip than one in the US. Apply the same to caches, queues, and storage.
- **Short timeouts.** Some servers are slow or dead. Set a maximum wait; if a host does not answer in time, drop that job and move on, do not let one slow site stall a worker.

### Deep dive 4: Robustness, and scaling the whole thing out

To hit the throughput and survive the web's hostility, the crawler runs as a fleet, and the fleet needs to grow, shrink, and tolerate failures without dropping or duplicating work.

![Distributed crawl: seed URLs enter a consistent-hashing layer that partitions the URL space by host across crawler nodes 1, 2 and 3; each node runs its own frontier-to-downloaders-to-parser loop and writes to a shared sharded-and-replicated content store and a URL and state store](https://mermaid.ink/img/Zmxvd2NoYXJ0IFRECiAgU2VlZChbU2VlZCBVUkxzXSkgLS0+IFJpbmdbQ29uc2lzdGVudCBoYXNoaW5nPGJyLz5wYXJ0aXRpb24gVVJMIHNwYWNlIGJ5IGhvc3RdCiAgUmluZyAtLT4gTjFbQ3Jhd2xlciBub2RlIDFdCiAgUmluZyAtLT4gTjJbQ3Jhd2xlciBub2RlIDJdCiAgUmluZyAtLT4gTjNbQ3Jhd2xlciBub2RlIDNdCiAgc3ViZ3JhcGggTm9kZVtJbnNpZGUgZXZlcnkgY3Jhd2xlciBub2RlXQogICAgZGlyZWN0aW9uIExSCiAgICBORltVUkwgRnJvbnRpZXJdIC0tPiBORFtIVE1MIERvd25sb2FkZXJzPGJyLz5tYW55IHRocmVhZHNdIC0tPiBOUFtQYXJzZXIgcGx1cyBzZWVuIGNoZWNrc10gLS0+IE5GCiAgZW5kCiAgTjEgLS4tPiBOb2RlCiAgTjIgLS4tPiBOb2RlCiAgTjMgLS4tPiBOb2RlCiAgTjEgLS0+IFN0b3JlWyhDb250ZW50IFN0b3JhZ2U8YnIvPnNoYXJkZWQgYW5kIHJlcGxpY2F0ZWQpXQogIE4yIC0tPiBTdG9yZQogIE4zIC0tPiBTdG9yZQogIE4xIC0tPiBVU3RvcmVbKFVSTCBhbmQgc3RhdGUgc3RvcmU8YnIvPmZyb250aWVyIHBsdXMgVVJMIFNlZW4pXQogIE4yIC0tPiBVU3RvcmUKICBOMyAtLT4gVVN0b3JlCiAgY2xhc3NEZWYgc2VlZCBmaWxsOiMyNTYzZWIsc3Ryb2tlOiMxZTQwYWYsY29sb3I6I2ZmZmZmZjsKICBjbGFzc0RlZiBib3ggZmlsbDojZWZmNmZmLHN0cm9rZTojMjU2M2ViLGNvbG9yOiMxZTNhOGE7CiAgY2xhc3NEZWYgcmluZyBmaWxsOiNmZWY5YzMsc3Ryb2tlOiNjYThhMDQsY29sb3I6IzcxM2YxMjsKICBjbGFzc0RlZiBzdG9yZSBmaWxsOiNkYmVhZmUsc3Ryb2tlOiMxZTQwYWYsY29sb3I6IzFlM2E4YTsKICBjbGFzcyBTZWVkIHNlZWQ7CiAgY2xhc3MgTjEsTjIsTjMsTkYsTkQsTlAgYm94OwogIGNsYXNzIFJpbmcgcmluZzsKICBjbGFzcyBTdG9yZSxVU3RvcmUgc3RvcmU7Cg==?type=png&bgColor=FFFFFF)

The load-splitting mechanism is **consistent hashing**: hash each host to a point on a ring and assign it to the nearest crawler node, so adding or removing a node only reshuffles a small slice of hosts instead of remapping everything. That is what lets the fleet grow and shrink mid-crawl. We won't re-derive the ring here, it is covered in [what is consistent hashing](/what-is-consistent-hashing); the point is it is *why* the distributed crawler survives node churn. Hashing by *host* (not by full URL) also keeps a host's URLs together on one node, which is exactly what the politeness back-queues need.

The rest of robustness is unglamorous but is what an interviewer wants to hear:

- **Save crawl state.** Periodically checkpoint the frontier and seen-sets to durable storage, so a crashed node restarts from where it was instead of re-crawling from seeds.
- **Handle every exception.** Bad HTML, connection resets, malformed URLs, a 500 from the server, catch and log, never let one page crash a worker.
- **Validate data.** Check content types and sizes before trusting a response.

### Detecting and avoiding problematic content

Three specific hazards worth naming, because the web is adversarial:

- **Duplicate content** (~30% of the web). Handled by "Content Seen?", hash each page and compare fingerprints:

```typescript
import { createHash } from "node:crypto";

// A page's fingerprint: same bytes to same hash. Store fingerprints, not full HTML, for the dedupe check.
class ContentSeen {
  private fingerprints = new Set<string>();

  isDuplicate(html: string): boolean {
    const fp = createHash("sha256").update(html).digest("hex");
    if (this.fingerprints.has(fp)) return true;
    this.fingerprints.add(fp);
    return false;
  }
}
```

  At web scale that `Set` is itself huge and would be sharded across machines or backed by a bloom filter, same trade-off as "URL Seen?", but the idea is exactly this.

- **Spider traps.** A page (or generated tree of pages) that loops a crawler forever, for example an infinitely deep calendar or `.../foo/bar/foo/bar/foo/bar/...`. There is no perfect automatic detector; practical defenses are a **max URL length**, a **max crawl depth per host**, and flagging hosts that produce an absurd number of URLs for manual review and blocklisting.
- **Data noise.** Ads, spam pages, boilerplate, low-value content. Filter what you can; a search-index crawler often runs an anti-spam scoring step to skip junk and save crawl budget.

## Follow-up questions to expect

**"A single 'URL Seen?' service becomes a hot spot at billions of URLs, what do you do?"** Shard it, partition the URL space (by host hash) so each shard owns a slice of the seen-set, and front each shard with a bloom filter so the common case, "yes, seen it", never touches the underlying table. It is the same consistent-hashing partition the crawler nodes already use.

**"How do you keep the crawl fresh without re-crawling a billion pages every cycle?"** Re-crawl by priority and observed change rate. Track each page's update history; pages that change often and matter (news homepages) get short re-crawl intervals, static archived pages get long ones. Feed re-crawl URLs back through the Prioritizer so freshness reuses the priority machinery instead of a separate system.

**"A viral site is flooding one back queue while others sit idle, hot-host problem."** The single-worker-per-host rule caps how fast any one host is crawled, so a hot host cannot starve others of *download* capacity, but its own back queue can grow unbounded. Cap the per-host queue length and shed or deprioritize the overflow; you rarely need every URL from one runaway host immediately.

**"What if a crawler node dies mid-crawl?"** Because state is checkpointed and hosts are assigned by consistent hashing, the dead node's hosts get reassigned to its neighbors on the ring, which reload the last checkpoint and continue. You lose at most the work since the last checkpoint, and duplicate a little, which the "URL Seen?" and "Content Seen?" checks absorb.

**"The pages use JavaScript to build their links, your parser sees an empty shell."** Right, static HTML parsing misses client-rendered links. The fix is a **rendering step**: run the page in a headless browser (server-side / dynamic rendering) to execute the JavaScript, then parse the rendered DOM. It is far more expensive per page, so you apply it selectively, only to sites known to need it.

## Wrap up

I started from a three-line loop and let the numbers break it, one line at a time: one billion pages a month forced a distributed downloader; 30 petabytes forced sharded storage; billions of URLs forced a bloom-filter "URL Seen?"; 30% duplicates forced "Content Seen?"; and the web's own structure, most links staying on one host, forced the two-layer URL Frontier that is the real heart of the design. The frontier is where I would spend the most interview time: front queues for priority, back queues for politeness, disk-backed with memory buffers.

The bottlenecks to name out loud: DNS resolution (cache it), the "URL Seen?" hot spot (shard and bloom-filter it), and any single host that is either fragile (be polite) or runaway (cap its queue). The next scale curve is the data layer, at 30 PB you lean hard on replication and sharding, which is a topic of its own ([how to scale databases](/how-to-scale-databases)), plus JavaScript rendering and anti-spam filtering to stop wasting crawl budget on junk.

The one-line recap: a web crawler is a distributed BFS over the web graph, and every hard part is a defense, against re-fetching, against duplicates, against rudeness, and against a web that is actively trying to trap you.