Common Crawl

Executes monthly crawl cycles capturing 3-5 billion web pages using the CCBot crawler agent, storing raw HTTP responses, headers, and page content in WARC (Web ARChive) format on AWS S3. Respects robots.txt and maintains an opt-out registry to exclude domains from crawling. Each monthly snapshot becomes a permanent archive layer, accumulating 300+ billion pages across 15+ years of operation.

Provides CDXJ (Capture inDeX JSON) indices that map URLs to their locations within WARC files, enabling random access to specific crawled pages without scanning entire archives. The index structure stores URL metadata and WARC file offsets, allowing efficient retrieval of individual pages from petabyte-scale datasets. Users query the index to locate a URL, then fetch only the relevant WARC segment from S3.

Provides a columnar index structure (format and technical details unknown from documentation) that enables efficient filtering and aggregation across crawl metadata without accessing raw WARC content. Allows queries on metadata dimensions like domain, content type, HTTP status codes, and capture timestamps. Designed for analytical workloads that need statistics or filtered subsets of the crawl without full content retrieval.

Extracts domain-level link graph from crawl data, capturing which domains link to which other domains and backlink relationships. Produces graph data (format unknown) representing the web's connectivity structure. Enables analysis of domain authority, link patterns, and web topology without processing raw page content. Referenced as 'BacklinkDB' in documentation but technical details not provided.

Stores all crawled web content in WARC (Web ARChive) format on AWS S3 public buckets, enabling distributed access without centralized bottlenecks. WARC is the ISO 28500 standard for web archival, containing HTTP requests, responses, headers, and payloads in a sequential record format. S3 storage provides global availability, parallel download capability, and HTTP range request support for partial file retrieval. Users access files directly via S3 API or HTTP without intermediary services.

Implements crawl exclusion mechanisms respecting robots.txt directives and a maintained opt-out registry where domain owners can request exclusion from future crawls. CCBot crawler agent checks robots.txt before crawling and consults the opt-out registry to avoid capturing content from domains that have requested exclusion. Provides a submission mechanism (details unknown) for domains to register opt-out requests.

Provides integration with Hugging Face Hub enabling discovery and download of Common Crawl data through the Hugging Face ecosystem. Specific integration details, API format, and available datasets unknown from documentation. Allows researchers to access Common Crawl data through familiar Hugging Face tools and interfaces rather than direct S3 access.

Provides community support infrastructure including a mailing list archive, Discord community channel, and FAQ section addressing common questions about data access, format, and usage. Enables peer-to-peer support and knowledge sharing among researchers and practitioners using Common Crawl. Blog with examples provides practical guidance on common tasks.

Serves as the primary raw data source for downstream dataset creation pipelines including C4, The Pile, RedPajama, FineWeb, and Dolma. These datasets apply text extraction, deduplication, filtering, and quality curation on top of Common Crawl's raw WARC archives to produce cleaned, deduplicated text suitable for language model training. Common Crawl provides the petabyte-scale raw material; downstream projects handle cleaning and curation.