warc2zip examples

Small, self-contained WARC files for trying out and testing warc2zip.

Files

The file names follow Common Crawl's repackage convention, <FLAVOUR>-REPACKAGE-<CRAWL-ID>.warc.gz, the same scheme used for e.g. EOT-2024-REPACKAGE-CC-MAIN-2024-42-GOV-000000-001.warc.gz in the End of Term Archive.

warc2zip outputs

For each WARC above, the ZIP archives produced by warc2zip are also provided, so the expected output can be inspected without running the tool. Three variants exist per WARC, matching the tool's output modes:

• .zip — the default flat format (--format flat): one payload file per response record, named 1000000.html, 1000001.html, … directly under the top-level directory, plus the CSV/JSONL header tables.
• .metadata-only.zip — flat format run with --metadata-only: the header tables and warcinfo only, no payloads.
• .sidecar.zip — the sidecar format (--format sidecar): payloads grouped into one directory per host, with the WARC and HTTP headers of every record written as small text files next to each payload (see Sidecar layout). The header tables are still generated.

Inside each ZIP, everything sits under a single top-level directory derived from the WARC-Filename of the source WARC plus a conversion timestamp and short hash, e.g. homepages_CC-MAIN-2026-21_20260907T143504_b3c3/ (see Things to know).

Sidecar layout

In the .sidecar.zip files each capture is placed in a directory named after the host of its WARC-Target-URI, and every WARC record belonging to the capture is unpacked into text files named by appending a suffix to the payload file name. Headers are kept in raw Name: value form. For a capture from the 500-record file, which retains request and metadata records, this gives seven entries:

CC-MAIN-20260618163205-20260618193205-00999_20260908T123325_2c87/
├── 010mes.com/
│   ├── 1000000.html                        payload
│   ├── 1000000.html.response.warc          WARC headers of the response record
│   ├── 1000000.html.response.http          HTTP response headers
│   ├── 1000000.html.request.warc           WARC headers of the request record
│   ├── 1000000.html.request.http           HTTP request line and headers
│   ├── 1000000.html.metadata.warc          WARC headers of the metadata record
│   └── 1000000.html.metadata.warc-fields   body of the metadata record (fetchTimeMs, charset, languages, …)
├── acc-ern.tul.cz/
│   ├── 1000041.pdf
│   └── …
├── warcinfo.warc, warcinfo.warc-fields
├── manifest.jsonl, manifest.csv
└── *.csv header tables (same set as the flat format)

The two cdx_toolkit repackages contain response records only, so their sidecar ZIPs carry just .response.warc and .response.http per payload; the request_*.csv and metadata*.csv tables are still written but hold only a header row.

Using them with warc2zip

Installation, usage and ready-made commands for each of these files are documented in the warc2zip repository — see in particular WARC examples for testing. warc2zip can read the files straight from the bucket URLs above; no download is required.

The WARC files are ordinary .warc.gz files and also work with any other WARC tooling.

File details

500_RECORDS-REPACKAGE-CC-MAIN-2026-30.warc.gz

The first 500 captures of a regular Common Crawl segment WARC (CC-MAIN-20260618163205-20260618193205-00999.warc.gz), kept exactly as the crawler wrote them: each capture consists of a request, a response and a metadata record, preceded by the original warcinfo record. This is the one to use for exercising request and metadata record handling.

Record counts: 1 warcinfo, 501 request, 500 response, 500 metadata.

warcinfo:

isPartOf: CC-MAIN-2026-25
publisher: Common Crawl
description: Wide crawl of the web for June 2026
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-233
software: Apache Nutch 1.21 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.7-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/

Note that the retained warcinfo record reports isPartOf: CC-MAIN-2026-25, the crawl the source segment belongs to.

HOMEPAGES-REPACKAGE-CC-MAIN-2026-21.warc.gz

Homepage captures (site roots such as https://example.com/, occasionally /index.html and the like) extracted from CC-MAIN-2026-21 with cdx_toolkit. Only response records are included; the matching request and metadata records are not.

Record counts: 1 warcinfo, 35,700 response.

warcinfo:

software: pypi_cdx_toolkit/0.9.40.dev89+g53a7ef76c
isPartOf: CC-MAIN-2026-21
description: Repackage of CC-MAIN-2026-21 containing only response records of homepages
format: WARC file version 1.0
creator: Common Crawl Foundation <https://commoncrawl.org>
operator: Malte Ostendorff <mailto:malte@commoncrawl.org>

IS_US_FEDERAL-REPACKAGE-CC-MAIN-2025-13.warc.gz

Captures from hosts classified as US federal government, extracted from CC-MAIN-2025-13 with cdx_toolkit as part of the End of Term Archive effort. Only response records are included.

Record counts: 1 warcinfo, 35,700 response.

warcinfo:

software: pypi_cdx_toolkit/0.9.40.dev91+ga04800ea0
isPartOf: CC-MAIN-2025-13
description: Repackage of CC-MAIN-2025-13 containing only response records of US federal government hosts
format: WARC file version 1.0
creator: Common Crawl Foundation <https://commoncrawl.org>
operator: Malte Ostendorff <mailto:malte@commoncrawl.org>

Things to know

• WARC-Filename still carries the original file names. The two cdx_toolkit repackages were renamed after they were written, so their warcinfo records say WARC-Filename: homepages_CC-MAIN-2026-21.warc.gz and WARC-Filename: is_us_federal_CC-MAIN-2025-13.warc.gz; the 500-record file names its source segment, CC-MAIN-20260618163205-20260618193205-00999.warc.gz. Tools that derive names from WARC-Filename will use those values.
• manifest.* records bare file names in every format. The filename column is 1000000.html also in the .sidecar.zip files, where the entry actually lives at <host>/1000000.html; the host directory has to be derived from warc_target_uri when joining the manifest to ZIP entries.
• Each ZIP was produced by a separate run, so the conversion timestamp and hash in the top-level directory differ between the .zip, .metadata-only.zip and .sidecar.zip of the same WARC. The header tables and manifests inside are otherwise identical across the three variants.
• The US federal repackage is dominated by a single host. Its 35,700 captures come from only six hosts, which is why its sidecar ZIP has almost everything under www.voanews.com/.
• The content is a redistribution of Common Crawl data and is subject to the Common Crawl Terms of Use.