# RRLS v0.10 Research Report

## What changed

v0.10 is a stricter research build rather than a compression-ratio showcase.

It adds:

- content-digest deduplication before vocabulary learning;
- per-host sampling caps;
- deterministic hash-shuffled training order so early corpus domains cannot monopolize the vocabulary;
- host-disjoint validation;
- held-out-only MDL accounting;
- frozen compiler transfer to a different corpus with **zero learning**;
- exact host and content-overlap checks between corpora;
- resumable WET shard downloads using HTTP Range;
- per-document zlib/LZMA baselines;
- a hybrid `RRLS tokens + entropy compressor` baseline;
- an operator-market module that is allowed to choose RAW or a standard compressor when RRLS loses;
- a cross-crawl evaluation CLI.

## Failure found by running the engine over itself

A 52.88 MB heterogeneous external-memory run initially produced only **0.563x** total MDL compression: worse than raw bytes.

The failure was vocabulary starvation. SQLite supplied training documents in insertion order and the finite phrase dictionary filled on early domains. Later domains never got a fair chance to create vocabulary.

v0.10 derives a deterministic `order_key` from each content digest and trains in that indexed pseudo-random order. Same corpus family, same model class, finite dictionary:

- before fix: 0.563x;
- after fix: 2.003x held-out at L4;
- zlib9 held-out baseline: 3.107x.

The fix therefore dramatically changes RRLS behavior, but RRLS still loses to zlib on pure compression. That is a useful negative result.

## Real uncontrolled local-data run

To avoid relying only on generated corpora while live Common Crawl binaries are blocked in this execution environment, v0.10 was also run on **48.17 MiB of actual installed system data** from the runtime: Python source/library files, system documentation and icon/image assets.

- documents: 6,240
- pseudo-host groups: 1,128
- held-out documents: 3,360
- best RRLS held-out MDL: **1.123x** at L4
- RRLS-token + zlib hybrid: **1.273x**
- zlib9 per-document baseline: **1.753x**

Again, standard compression wins. This rejects the hypothesis that the current phrase vocabulary is already a superior general compressor.

## Frozen-vocabulary transfer

A v0.8 hierarchy trained on one 11 MiB heterogeneous corpus was frozen and applied to an independently generated 11 MiB corpus with different seeds and **no vocabulary updates**.

- frozen RRLS hierarchy including dictionary cost: **2.026x**;
- independent per-document LZMA: **2.984x**;
- LZMA allowed to train across the concatenated target corpus: **4.067x**.

Adding an entropy coder to the learned tokens improved one large generated held-out run to **2.200x**, but still did not beat the ordinary compressor baseline.

The scientifically interesting quantity is therefore not yet raw compression. It is **transferable structural vocabulary under representation changes**.

## Leakage gate

During development, a cross-corpus test used disjoint hostnames but accidentally regenerated identical page bodies. The new digest-overlap test detected **1,200 / 1,200 exact content overlaps** and invalidated the experiment. After changing both hosts and page bodies, overlap was zero and frozen transfer still compressed, but below zlib.

This is now a mandatory cross-crawl gate:

1. host overlap;
2. exact document digest overlap;
3. frozen-vocabulary held-out cost;
4. standard compressor baseline.

## Position relative to existing research

Hierarchical phrase grammar discovery is not new by itself. SEQUITUR discovers recursive grammar structure, Re-Pair recursively replaces frequent pairs, and Minimum Description Length formalizes the idea that useful regularities permit shorter descriptions. Static trained dictionaries are also standard in compressors such as zstd.

RRLS therefore should **not** claim novelty merely because it recursively learns phrase vocabulary.

The research hypothesis worth testing is narrower:

> Search a portfolio of reversible representations and relation operators; retain only relations whose complete description reduces held-out cost; freeze them; test whether they transfer to unseen hosts, future crawls and different data domains; then use the validated relations themselves as the next alphabet and repeat.

Evidence for something genuinely new would require the relational hierarchy to discover compact transferable operators that are not explained by established grammar/dictionary compression, or to solve prediction/reconstruction tasks substantially better under equal description/compute budgets.

## Common Crawl experiment

The live Common Crawl downloader remains in the package. The current binary sandbox cannot fetch WET objects, so no Common Crawl compression number is reported from this environment.

The correct next empirical protocol is:

- train: `CC-MAIN-2026-34`, random WET shards, host capped + content deduplicated;
- within-crawl test: unseen August hosts;
- frozen cross-crawl test: a separate crawl such as `CC-MAIN-2026-30`;
- reject runs with exact document leakage;
- compare RAW, RRLS layers, RRLS+entropy coding, zlib/LZMA/zstd-style baselines;
- inspect whether higher RRLS tokens become more cross-language/cross-host than lower tokens.

`rrls_crosscrawl_eval.py` implements the frozen-corpus half of this protocol.