Internet Archive vs Common Crawl vs web archive

We queried the CC-MAIN-2026-30 Common Crawl index for theguardian.com and got 185 records. Not one is an article. Every record is the site’s own robots.txt or a redirect to it, and that file disallows CCBot. The permission check is the only thing the crawl kept.

The Internet Archive’s Wayback Machine and Common Crawl both hold copies of pages that no longer exist, and they answer different questions. The Wayback Machine replays a single URL as it looked on a date you pick. Common Crawl hands you billions of pages as bulk text, with no replay. A web archive is the category both belong to rather than a third product. The same category holds national web archives, subscription services like Archive-It, on-demand capture sites like archive.today, and WARC files you write yourself. The two services answer questions about the past; neither collects a named set of sites going forward.

TL;DR

• CCBot does not execute JavaScript. A client-rendered page arrives as the shell.
• Query Common Crawl first for bulk work and Wayback for the misses; reverse it for a known URL list.
• A CDX digest is the payload SHA-1, so collapse=digest and collapse=timestamp diff in the index before you fetch. Common Crawl ignores collapse.
• Design a Wayback CDX client at 24 requests per minute, 80% of the shared 30/min pool.
• Neither publishes a coverage guarantee or an SLA, and neither service collects a named set of sites going forward. Measure your domains first.

What the Internet Archive and Common Crawl each store

A Wayback capture is addressable as web.archive.org/web/<timestamp>/<url>, and the same URL can carry thousands of captures across three decades. Default replay rewrites links and injects a toolbar so the page renders in a browser.

Each Common Crawl monthly crawl is a set of WARC files plus derived text and metadata files, published to a public bucket. An index tells you which byte range of which file holds a given URL.

CC-MAIN-2026-30 is announced at 2.14 billion pages and 364 TiB of uncompressed content across 40.5 million hosts, crawled over 18 days. Common Crawl puts the whole corpus at over 300 billion pages and describes the rate as 3 to 5 billion new pages a month, though the last 18 published crawls have all come in under 3 billion. Size a job from a single crawl.

Common Crawl’s FAQ says: “Currently, JavaScript is not executed and Cookies are not used”. CCBot stores the raw HTTP response, so a client-rendered page arrives as its shell. If the pages you care about build their content in the browser, check one capture before planning around Common Crawl.

Each crawl ships five parallel file sets of 100,000 files each. WARC carries the full response, WAT carries extracted metadata and links, WET carries plain text only, and two more carry the robots.txt captures and the non-200 responses. A link-graph job reading WARC moves far more bytes than it needs.

Common Crawl makes no promise that it revisits any page, so many URLs across many dates is a question neither service answers.

Common Crawl’s FAQ states the sampling policy: the dataset “is a sample of the web, and we do not generally archive any entire website but a randomly selected subset of it”. The mechanism is documented in a Common Crawl engineering talk rather than the FAQ: domain-level harmonic centrality ranks define a budget for how many URLs each domain gets, so well-linked domains get a larger budget and low-ranked domains may get none.

The Wayback Machine has no full-text search of archived page content. Its own help page says the Internet Archive hopes “to implement a full text search engine at some point in the future”. The site search matches site metadata, not the text inside archived pages. Common Crawl gives you the text but no replay, so a page reconstructed from a WET file will not look like the original.

What a domain lookup returns

We ran the same query against five domains, with prefix matching.

The Guardian’s 185 records resolve to just two URLs, the robots.txt and its http redirect, re-fetched through the crawl. A script counting index records would call all five covered and be wrong on four. Of github.blog‘s 4,651 content records, 3,647 answered 200.

Match type changes the answer. SURT canonicalization folds www. into the bare host, but a domain/ prefix match reaches no other subdomain: cnn.com/ returned 1 record while .cnn.com returned 11, and nytimes.com/ returned 1 against 2 for *.nytimes.com. Every record was a robots.txt fetch under either shape, but the counts are a property of the query as much as of the crawl. Report your match type whenever you report a number like this.

The same query dates the block roughly if you run it against older crawl IDs. The query has a blind spot: it sees what a publisher wrote in robots.txt, not a block applied at the network edge or a later opt-out, so it reads as a floor rather than a count. Our robots.txt guide for scraping covers how the agent groups resolve.

Per-crawl size has been falling as well. The Common Crawl published crawl-size statistics show the monthly page count dropping from 3.031 billion in CC-MAIN-2025-05 to 2.149 billion in CC-MAIN-2026-30, down 29.1%. Across the 29 crawls from CC-MAIN-2024-10 to CC-MAIN-2026-30, the 10 crawls of 2024 average 2.69 billion pages, the 12 crawls of 2025 average 2.53 billion, and the 7 crawls published in 2026 through July average 2.15 billion.

Nothing here identifies a cause: crawl size moves with infrastructure, budget, and scheduling inside Common Crawl as much as with anything publishers do. The decline does mean that a corpus sized from a 2024 crawl overestimates a 2026 one by about a quarter, and that per-crawl counts do not add. Consecutive crawls re-fetch much of the same frontier, so summing them without deduplicating on urlkey and digest inflates the estimate again.

Check your own domains

The script counts content records against robots.txt fetches for the newest published crawl, using the same domain/* prefix match as the table. It walks the index a page at a time, rejects a page whose truncation splits a record rather than counting part of one, and keeps going when a domain never answers cleanly. The two domains below take about 70 seconds, nearly all of it the deliberate sleeps:

import http.client, json, time, urllib.error, urllib.parse, urllib.request

DOMAINS = ["theguardian.com", "github.blog"]
# Use your own contact address. One shared UA string arriving from many callers
# is the string an operator blocks.
UA = {"User-Agent": "coverage-check/1.0 ([email protected])"}

def latest_crawl():
    """Common Crawl ships roughly monthly. Hardcoding an ID measures a stale crawl
    and returns a number that looks current."""
    url = "https://index.commoncrawl.org/collinfo.json"
    info = urllib.request.urlopen(urllib.request.Request(url, headers=UA), timeout=60).read()
    return json.loads(info)[0]["id"]        # newest first

CRAWL = latest_crawl()
INDEX = f"https://index.commoncrawl.org/{CRAWL}-index"
print(f"crawl: {CRAWL}")

def fetch(params, tries=3):
    url = f"{INDEX}?{urllib.parse.urlencode(params)}"
    for attempt in range(tries):
        wait = None
        try:
            return urllib.request.urlopen(
                urllib.request.Request(url, headers=UA), timeout=120).read()
        except urllib.error.HTTPError as e:
            if e.code == 404:
                return None               # this crawl never captured the domain
            if e.code not in (429, 500, 502, 503, 504):
                raise
            wait = e.headers.get("Retry-After")   # the server's own number beats ours
        except (urllib.error.URLError, http.client.IncompleteRead, TimeoutError):
            pass                          # dropped connection
        time.sleep(int(wait) if wait and wait.isdigit() else 5 * 2 ** attempt)
    return b""

def coverage(domain):
    """(robots, content) for one domain, or None if the index never answered cleanly."""
    query = {"url": f"{domain}/*", "output": "json", "pageSize": 1}
    head = fetch({**query, "showNumPages": "true"})
    if head is None:
        return 0, 0
    if not head:
        return None
    robots = content = 0
    for page in range(json.loads(head)["pages"]):
        body = fetch({**query, "page": page})
        if not body:
            return None
        # CDXJ is one JSON object per line. Use split, not splitlines(): splitlines()
        # breaks on several characters that are not newlines, which cuts records in half.
        for line in body.split(b"\n"):
            if not line.strip():
                continue
            try:
                record = json.loads(line)
            except json.JSONDecodeError:
                return None               # the page arrived incomplete
            if record.get("url", "").rstrip("/").endswith("/robots.txt"):
                robots += 1
            else:
                content += 1
        time.sleep(5)
    return robots, content

for domain in DOMAINS:
    result = coverage(domain)
    if result is None:
        print(f"{domain}: no clean response")
    else:
        robots, content = result
        print(f"{domain}: {robots + content} records, {robots} robots.txt, {content} content")
    time.sleep(20)

It prints:

crawl: CC-MAIN-2026-30
theguardian.com: 185 records, 185 robots.txt, 0 content
github.blog: 4700 records, 49 robots.txt, 4651 content

Against CC-MAIN-2026-30 the two domain lines reproduce the table above. A newer crawl returns different counts; what should hold is the shape, robots-only for theguardian.com and content-bearing for github.blog. Records without content pages mean the crawl holds no page content from that domain.

How to query each source, and what the round trip costs

Both services expose a CDX index, but the response formats differ: the Wayback CDX server returns a JSON array of arrays, Common Crawl’s output=json returns CDXJ, one JSON object per line.

The Wayback CDX server returns one row per capture, and filtering server-side keeps the response small enough to page through:

curl "https://web.archive.org/cdx/search/cdx?url=example.com&output=json&from=2026&limit=3&fl=timestamp,original,statuscode"

That returns:

[["timestamp","original","statuscode"],
["20260101000936","http://www.example.com/","200"],
["20260101002937","https://example.com/","200"],
["20260101004445","https://example.com/","200"]]

To parse an archived page instead of rendering it, request the raw capture. Adding id_ after the timestamp returns the original bytes, with no toolbar and no rewritten links:

curl -sL "https://web.archive.org/web/2026id_/https://example.com/" -o raw.html
curl -sL "https://web.archive.org/web/2026/https://example.com/"    -o rewritten.html

The rewritten copy carries injected references to web.archive.org, which can break selectors that anchor on DOM structure.

Beyond match type, two fields in a CDX row and one query parameter decide what a query matches, whether you can detect change, and what it costs.

urlkey is the SURT form of the URL. https://github.blog/ indexes as blog,github)/: host reversed, then path. Prefix and range queries operate on that key rather than on the URL you typed, which is why www. handling and query-string order decide whether a query matches anything.

digest is the payload SHA-1, base32-encoded, and it is the value the WARC record carries as WARC-Payload-Digest. A hex digest will not match it. Change detection is therefore an index operation: two rows sharing a digest have the same payload bytes.

collapse turns that into the query you actually want. Asking the Wayback CDX server for every example.com capture in calendar 2024 returns 130,115 rows. The same query with collapse=timestamp:8 returns 366, one per day: timestamps are YYYYMMDDhhmmss, so truncating to eight digits collapses on the date. With collapse=digest it returns 17,448, one per change between adjacent captures. Common Crawl’s index accepts the parameter and ignores it, returning the same rows either way.

Pagination through page and pageSize counts index blocks rather than result rows, and block size varies: two adjacent blocks on one query returned 2,032 and 4,726 records. Querying the Wayback CDX server for bbc.com with matchType=domain and pageSize=1 reported 28,257 pages.

A page can come back empty simply because your filters matched nothing in those blocks, which does not mean the result set has ended. The default pageSize is 1, and the documentation names no maximum.

Common Crawl’s index paginates the same way, with three differences: showNumPages returns JSON rather than a bare number, the default pageSize is 5 rather than 1, and its blocks are a different size. Size a loop from showNumPages rather than from an assumed block size, and ask for the page count at the pageSize you intend to use, because the count changes with it.

There are two ways to iterate. The default is limit mode, where limit and a resume key walk the result set; paged mode uses page and pageSize instead. Paged mode exists so the page count is knowable in advance and the work fans out across workers. Limit mode usually makes fewer requests overall, but on a rarely captured URL across a wide date range it can scan far enough to time out.

Common Crawl inverts the fetch step, and the payoff is that pulling one page costs a single range request instead of a file download. The index gives you a filename, a byte offset, and a length, so an HTTP range request pulls back one record. Because CDXJ puts one object per line, the code below reads one line rather than parsing the whole body:

import gzip, json, urllib.error, urllib.parse, urllib.request

url = "https://example.com/"
crawl = "CC-MAIN-2026-30"
q = urllib.parse.quote(url, safe="")

# Common Crawl asks API clients for a properly formed UserAgent. Use your own.
ua = {"User-Agent": "coverage-check/1.0 ([email protected])"}

# Step 1: ask the index which WARC file and byte range holds this page.
index = f"https://index.commoncrawl.org/{crawl}-index?url={q}&output=json&limit=1"
try:
    req = urllib.request.Request(index, headers=ua)
    rec = json.loads(urllib.request.urlopen(req, timeout=60).read().split(b"\n")[0])
except urllib.error.HTTPError as e:
    if e.code != 404:
        # 502, 503 and 504 all mean try again later, not "not captured".
        # Treating them as a miss under-reports coverage.
        raise
    # A URL this crawl never captured returns 404. That is the normal case for a
    # blocked domain, not an error in your code.
    raise SystemExit(f"{url} is not in {crawl}")

# Step 2: fetch only those bytes.
offset, length = int(rec["offset"]), int(rec["length"])
warc = "https://data.commoncrawl.org/" + rec["filename"]
req = urllib.request.Request(warc, headers={**ua, "Range": f"bytes={offset}-{offset + length - 1}"})
record = gzip.decompress(urllib.request.urlopen(req, timeout=60).read())

print(record.decode("utf-8", "replace")[:400])

That returns one gzipped WARC record carrying the response headers and the page HTML. Across 29 content records we sampled, compressed size ran 7 to 45 KiB, so the range request moves kilobytes out of a file of about 900 MiB. The same path works against any crawl the index server lists.

For work at corpus scale, Common Crawl’s URL Index ships the same index as Parquet with extra columns, published at s3://commoncrawl/cc-index/table/cc-main/warc/ and queryable with DuckDB locally or Athena in AWS. Common Crawl puts the columnar index at roughly 300 GB per monthly crawl and a full Athena scan at about $1.50 as of September 2025, so most filtered queries cost less. Data access itself is free through the AWS Open Data program, and data.commoncrawl.org needs no AWS account, though egress from your own cloud provider is still yours to pay.

Rate limits and quotas you will hit

Common Crawl’s FAQ says the CDX endpoint “is frequently abused and therefore heavily rate limited”. An HTTP 503 means slow down, and a temporarily blocked IP should wait 24 hours. The guidance also asks you to sleep between calls, avoid multiple threads from one IP, avoid proxy networks, and send a properly formed User-Agent, which its FAQ ties to RFC 9110. The proxy clause is the one worth dwelling on. The FAQ’s block and its threading guidance are both addressed at the IP, so spreading a job across more addresses works around the limit rather than respecting it, and the escalation is a block on the address rather than more 503s.

The Internet Archive publishes quotas for Save Page Now and effectively nothing for the read APIs. The Save Page Now 2 specification gives these limits:

The specification names no paid tier above these numbers and directs heavier users to email the Internet Archive. The specification gives the anonymous figure as both 2 GB and 500 MB, and its changelog lowered the authenticated figure to 4 GiB while the table still says 5 GB.

For the read APIs, the figures come from outside the Internet Archive. The maintainer of the wayback Python client wrote up a conversation with Internet Archive staff and encoded the outcome as the library’s defaults. The write-up is not an Internet Archive publication.

The /cdx/search/cdx and /web/timemap/ endpoints are now one service on the same servers, differing only in how they read the output parameter, and they share a single rate-limit pool. That pool is 30 requests per minute across both. The write-up does not say whether the counter is keyed to an address or an account, so a job fanned out across workers cannot assume the budget multiplies. The memento endpoint serves replay and draws on a separate pool.

The library encodes those limits as its defaults, 0.8 * 30 / 60 for CDX and 0.8 * 600 / 60 for replay, after Internet Archive staff asked that clients sit at 80% of the hard limits. That is 24 requests per minute rather than 30, and 480 for replay. Design against those, not the caps.

The availability endpoint at archive.org/wayback/available is a third service with no published limit of its own, so treat it as limited rather than free. Honor Retry-After when the response carries one. When it carries none, the wayback client pauses for 60 seconds before trying again.

Why archive.today captures cannot be verified on their own

archive.today renders pages in a browser at capture time and ignores robots.txt, so it holds pages the Wayback Machine does not. Citation workflows adopted archive.today for that coverage.

archive.today stores mutable HTML rather than WARC, so there is no payload digest and nothing inside the capture to check it against. Evidence was presented that its operators altered archived pages, and Wikipedia deprecated the site and added it to the spam blacklist. The closing statement leads with a different ground: it says the site embedded code that turned visitors’ browsers into a denial-of-service attack against a blog, and treats the altered snapshots as additional.

Re-verify anything you already cite from archive.today against a second source. The same page records that the code was still present in June 2026 at a reduced call rate, and tells anyone who needs the site to load it behind a content blocker.

National archives, formats, and replay tools

Arquivo.pt: the Portuguese national web archive runs a full-text search API that the Wayback Machine does not have. A GET to arquivo.pt/textsearch returns JSON carrying a linkToExtractedText per result, so you skip HTML parsing entirely. The API is free and needs no key, and the documented limit is 250 requests per 60 seconds from one IP, shared across full-text and URL search rather than budgeted separately. Exceeding that limit is documented as grounds for a permanent block, where Common Crawl’s block is temporary. Coverage is Portuguese-focused, and latency is uneven, from under 3 seconds to 18 across three single-result queries, so set the client timeout to 30 seconds.

WARC and WACZ: WARC is the archival container, standardized as ISO 28500:2017, which covers WARC 1.1. There is no WARC 1.2. WACZ packages WARC records with a CDX index so a browser can replay them without a server. A 1.2.0 draft is up, but wacz/latest still serves 1.1.1, so implement against 1.1.1.

Browsertrix: when you need captures you control, Webrecorder’s crawler produces WACZ files that replay in a browser through ReplayWeb.page with no server. Hosted plans start at $30 per month for the entry plan, and the crawler is open source if you would rather run it yourself.

How to choose

Start from the shape of the question: specific URLs, text in bulk, or a property neither service offers.

Use the Wayback Machine when the URL and the date are the point. Recovering a deleted page, proving what a site said before an edit, tracking one product page over time, or rebuilding a dead documentation set. No other general-purpose option matches its time depth on individual URLs, though a national archive can match it within its own scope.

If the use is evidentiary, budget for more than the capture. US courts have admitted Wayback captures, but a printout alone is rarely enough, because captures are not self-authenticating. An Internet Archive declaration is the route courts have accepted, and at rates published in early 2025 it runs $250 per request plus $20 for each URL, or $30 for URLs holding a downloadable file such as a PDF.

Use Common Crawl when you want many pages and do not care which. Language corpora, model pretraining, link-graph analysis, security research across many hosts, and any measurement where a rank-skewed sample is acceptable. Most of the large open pretraining corpora draw on it, including C4, RefinedWeb, FineWeb, and Dolma. If you want a corpus rather than the raw crawl, those derivatives and their licenses are worth reading first, and our survey of LLM training data sources is a starting point. Do not use it to guarantee coverage of a named site, because the sampling policy makes coverage unpredictable.

A study by Sichang Steven He and colleagues classified about 100,000 sites in Common Crawl and found 6.0% of them dominated by text generated with little human input. Among sites first seen in the first half of 2025, the share was 29.4%, up from 2.1% among sites first seen in late 2022.

Read the 6.0% and the 29.4% as likely undercounts. The authors report a strong negative correlation between their classifier’s accuracy and the benchmark score of the model that generated a site, so the share the classifier misses grows as generators improve.

Use a national or self-run archive when you need a property that neither service offers. Full-text search over archived content, captures you control end to end, or client-rendered pages that replay badly elsewhere.

archive.today is excluded on a different ground: a capture that can change after the fact cannot stand on its own as evidence.

For bulk work across many hosts, query Common Crawl first for breadth and cost, then fall back to the Wayback Machine for what the crawl missed. For a known URL list, go the other way: most Common Crawl lookups will miss, and every lookup spends a request against an index whose documented penalty for overuse is a 24-hour block. And when a domain matters to your work for the long run, run your own captures with a WARC-producing crawler in parallel, because that is a copy whose availability you control.

Running your own crawler costs storage, replay infrastructure, and a maintenance burden neither public service charges you for, so run it yourself where losing access would break something, and lean on the public archives or bought collection for the rest.

When you need collection instead of an archive

Neither service is built to collect a defined set of sites on a schedule. Save Page Now does collect on demand, and it runs at 7 captures a minute when authenticated, 3 when not.

All four news domains we queried had no content pages in CC-MAIN-2026-30. Neither publishes a coverage guarantee, an SLA, or a support commitment you can escalate against.

When you need named sites collected on a schedule, with coverage you define, that is a collection product rather than an archive. Crawling discovers what is out there; targeted collection returns known pages on demand. Our crawling vs scraping guide works through the difference.

Paying for collection does not buy an exemption from blocking. Publishers restrict commercial crawlers as well as archive crawlers; a vendor changes who carries the maintenance when a site changes its defenses.

Bright Data sells targeted collection: the Web Scraper API at $1 per 1,000 records, with the first 5,000 each month free, and ready-made datasets from $0.0025 per record on a $250 minimum. Both cover supported sites on an ongoing basis. Neither replaces the Wayback Machine, and the reason is a date range: pre-collected data reaches back days to months, not to 1996, so a question about what a page said in 2014 still goes to the archive.

Next steps

Run the coverage script on the domains you care about, then check the same domains in the Wayback CDX server for capture density.

Content records tell you whether bulk work is possible; capture density tells you whether URL-level history is possible. A healthy count is a reading rather than a commitment: the sampling policy promises no floor, and per-crawl totals have fallen about a quarter since 2024, so re-run the check before you depend on it.

Capture what matters now where losing access would break something, buy collection for the rest, keep what you capture in WARC so it stays portable, and treat both public archives as sources you query rather than storage you depend on.

Frequently asked questions

Can I use Common Crawl commercially?

The terms of use permit it and place the risk on you. Common Crawl caps its total liability at $100 and requires you to indemnify it for claims arising from using crawled content to develop, train, or deploy AI systems. Common Crawl recommends legal advice before any commercial use.

Can you download all Wayback Machine data?

Not in bulk. The Internet Archive publishes no bulk export of the general web archive, and access runs through rate-limited APIs. For research-scale analysis the Internet Archive offers ARCH, its compute platform, which is quote-only and has no published pricing.

What rate should I design a Wayback CDX client against?

Design against 24 requests per minute. The limit is 30 per minute, which the Internet Archive does not publish, shared across the CDX and timemap endpoints since they became one service, and the wayback client ships 80% of it as its default. Replay is a separate pool at 600 per minute, so design against 480 there.

Why does a Common Crawl domain lookup return records but no content?

Because those records are robots.txt fetches. CCBot requests robots.txt from every host it touches, and when a site disallows it, that request and any redirect to it are all the crawl retains. Classify returned URLs before concluding a domain is covered, and state your match type, because a prefix query and a domain query can return different counts.