# RRLS v0.9 — Common Crawl Web-Corpus Mode

## Why Common Crawl

Common Crawl is a free open repository of web-crawl data. The current Common Crawl
index at packaging time is `CC-MAIN-2026-34` (August 2026). Common Crawl reports that
the August archive contains about 2.14 billion web pages / 360 TiB uncompressed.

v0.9 adds a bounded, sequential Common Crawl WET sampler rather than attempting to
mirror the archive. WET records contain extracted page text.

## Experimental discipline

RRLS does **not** random-split pages.

It hashes the hostname and assigns the entire hostname to either train or test:

    split(host) = hash(host) mod 100

This matters because repeated navigation, CMS templates, cookie notices and page
chrome from one site would otherwise leak directly into held-out data. A vocabulary
item is therefore more interesting when it survives on unseen hosts.

WET language annotations are retained only as post-hoc domain labels. The compiler
still sees integer coordinates.

## External-memory architecture

The previous Universal Corpus Lab held its working corpus in memory. v0.9 uses SQLite:

- raw WET payloads are compressed and stored on disk;
- layer 1 is learned from training-host documents;
- the layer is frozen;
- every train/test document is encoded to uint32 token blobs;
- layer 2 learns only from the frozen layer-1 tokens;
- the process repeats;
- dictionary cost is charged in the MDL proxy;
- recursive expansion stops when the next layer no longer earns its cost.

This makes multi-gigabyte local experiments possible without requiring the entire
corpus to fit in RAM.

## Common Crawl UI

Run:

    launch_commoncrawl_lab.bat

or build:

    build_commoncrawl_lab.bat

The app exposes three presets:

- Probe — 256 MiB plaintext target
- Go ham — 4 GiB
- Monster — 20 GiB

All downloads are sequential by default. The program first downloads the small
`wet.paths.gz`, uniformly samples WET shards by seed, then streams conversion records
until the document or plaintext-byte cap is reached.

You can append multiple samples to the same SQLite corpus and then recompile it.

## Offline WET systems benchmark

Because the ChatGPT execution runtime used to package v0.9 does not expose arbitrary
binary Common Crawl objects to the local Python runtime, the network ingestion path
could not be truthfully benchmarked here against live WET shards.

The exact WET parser + SQLite + recursive compiler was instead tested end-to-end on a
locally generated WET-format archive:

- documents: 12,000
- hosts: 3,383
- raw plaintext: 13.54 MiB
- WET parsing/storage throughput: 42.40 MiB/s
- recursive compilation: 3.851 s
- best layer: L4
- best MDL proxy: 16.665x

The generated WET corpus is intentionally highly repetitive, so the compression ratio
is only a systems validation and must not be interpreted as a web-corpus result.

## What to inspect in a real run

The most important outputs are not just compression ratios:

1. **Best recursive depth** — does the vocabulary hierarchy stabilize?
2. **Host-held-out MDL** — does vocabulary still pay for itself away from the sites
   that created it?
3. **Cross-language tokens** — which exact phrases and structural relations survive
   across WET language labels?
4. **Vocabulary growth vs corpus growth** — does dictionary growth sublinearize?
5. **Opaque residual** — what percentage is still cheaper to leave raw?
6. **Layer transfer** — do higher tokens become more cross-domain than lower ones?
7. **Crawl-to-crawl persistence** — train on one monthly archive and test on another.

The last test is especially important. A candidate 'web language' should survive not
only unseen hosts but also a later crawl.

## Limitations

This is an experimental compressor/relation-discovery engine, not evidence of a
universal language of reality. Common Crawl contains duplicated content, generated
pages, spam, boilerplate, encoding artifacts and crawl-specific structure. Host-level
holdout removes one major leakage source but not all of them.

WET is text-only extracted content. Raw HTML, HTTP headers, images and other media
require WARC sampling/range-fetch support, which is the next obvious data layer.