Common Crawl Data Stored on a Hugging Face Bucket
4 hours ago
- Since April 2026, Common Crawl's crawl archive is accessible via Hugging Face Storage Bucket, in addition to AWS S3, offering HF ecosystem tools for easier data access.
- Selected crawls are mirrored in the HF bucket with the same directory layout as S3 (e.g., crawl-data/CC-MAIN-2026-17/); the bucket README lists available crawls.
- Users can access data via the HF CLI (e.g., `hf buckets list` and `hf buckets cp`), mount as local filesystem with hf-mount, or use the S3-compatible API for existing pipelines.
- WARC files can be read using the warcio library with fsspec and the hf:// protocol; code examples show iterating records and using the CLI.
- The URL Index (Apache Parquet format) is available on the HF bucket and can be queried with DuckDB, either via CLI or Python, enabling efficient analytical queries without full downloads.
- Download speed tests show S3-to-S3 (us-east-1) is fastest (up to 3,500 records/s), HF-to-HF reaches 1,500 records/s, but HF-to-edge clients are slower (<100 records/s) due to rate limiting; results are preliminary.
- Jupyter notebooks with examples are available in the cc-notebooks GitHub repository, and feedback can be provided via Discord or Google Group.