Common Crawl

Operates a distributed web crawler (CCBot) that systematically traverses 3-5 billion web pages monthly, capturing raw HTML, metadata, and response headers into WARC (Web ARChive) format files stored on AWS S3. The crawl respects robots.txt directives and maintains an opt-out registry for content exclusion. Each monthly snapshot is immutable and indexed for retrieval, creating a cumulative archive of 300+ billion pages spanning 15+ years of web history.

Provides CDXJ (Capture inDeX JSON) indices that map URLs to byte offsets within WARC files, enabling direct random access to specific pages without scanning entire archives. Queries specify a URL and optional date range, returning matching captures with metadata (HTTP status, content type, timestamp). This index layer abstracts away WARC file complexity and enables efficient lookup of historical versions of individual pages.

Publishes infrastructure status updates, known issues, and errata for crawls through a public status page and mailing list. Issues are documented with affected crawls, impact assessment, and workarounds. Status monitoring includes S3 availability, index health, and crawl progress. Errata tracking enables users to identify and work around data quality issues in specific crawls.

Operates a distributed web crawler (CCBot) that can be configured with custom crawl parameters including politeness delays, user-agent strings, robots.txt interpretation, and domain-specific crawl budgets. The crawler respects HTTP standards and robots.txt directives, with configurable behavior for handling redirects, timeouts, and errors. Crawl parameters are documented for each monthly release, enabling reproducibility and evaluation of crawl quality.

Provides columnar indices (format and query syntax unspecified in documentation) that enable structured queries across archive metadata without parsing WARC files. Queries can filter by domain, content-type, HTTP status, crawl date, and other fields, returning matching page metadata and offsets. This approach trades random-access flexibility for efficient bulk filtering and aggregation across billions of pages.

Extracts hyperlink relationships from crawled pages to construct a directed web graph showing which pages link to which other pages. This graph data is provided separately from raw page content, enabling analysis of link structure, PageRank-like metrics, and domain authority without parsing HTML. The extraction process identifies both internal (same-domain) and external (cross-domain) links.

Enables retrieval of any page version from the cumulative 300+ billion page archive spanning 2007-present, with monthly granularity. Users specify a URL and date range, and the system returns all captures of that page from matching crawls. This creates a time-series view of how individual pages evolved, including content changes, design updates, and deletion/resurrection events.

Exports raw web content in WARC (Web ARChive) format, a standardized container that bundles HTTP request/response pairs with metadata. Each WARC record includes the original HTTP status code, headers, response body (HTML, JSON, binary), and crawl metadata (timestamp, IP address, user-agent). WARC files are gzip-compressed and stored on S3, with indices enabling random access to specific records without decompressing entire files.

Publishes monthly snapshots of the web crawl on a documented schedule, with each release including 2-5 billion pages, comprehensive statistics (page counts, size, coverage by domain/TLD), and known issues (errata). Each crawl is assigned a unique identifier and published with metadata enabling reproducible research. The release process includes documentation of crawl parameters (user-agent, politeness delays, robots.txt compliance) and known limitations.

Respects robots.txt directives and maintains an opt-out registry allowing content creators to exclude their sites from crawling and archival. The CCBot crawler checks robots.txt before crawling each domain and honors disallow rules. Additionally, a public opt-out registry enables site owners to request retroactive removal from the archive. Compliance is enforced at crawl time (robots.txt) and archive time (opt-out registry).

Provides integration with Hugging Face Hub, enabling researchers to access Common Crawl data through the Hugging Face datasets library and export processed datasets directly to the Hub. This integration abstracts away S3 access complexity and enables one-line dataset loading in Python. Processed datasets (C4, The Pile, RedPajama, FineWeb, Dolma) are published on the Hub with documentation and usage examples.

Enables third-party researchers and organizations to build and publish extraction pipelines that transform raw Common Crawl WARC data into clean, deduplicated, filtered datasets suitable for model training. Major pipelines (C4, The Pile, RedPajama, FineWeb, Dolma) are published with open-source code, documentation, and reproducible builds. These pipelines handle deduplication, language filtering, quality scoring, and format conversion.