# RRLS v0.11 — Support-Gated Predictive Relations

## Status

v0.11 is a falsification/repair release. The goal of this loop was not to maximize a headline compression number; it was to make every proposed relational lift pay for itself on held-out data under a valid coding model.

The live Common Crawl WET objects remain unreachable from the ChatGPT execution container because its local runtime has no external DNS/binary-fetch path. Common Crawl itself is reachable through the web layer, and its official Hugging Face bucket exposes the crawl hierarchy, but the binary objects cannot be mounted into this Python sandbox. The same v0.11 code remains available in the Windows Common Crawl Lab for local live runs.

## Uncontrolled real-data corpus

The v0.11 stress corpus was built from real files already present in the execution runtime, not generated patterns:

- Python standard-library source/data
- system documentation
- icon/image assets
- MIME metadata
- native shared-library/binary data

Files were split into 64 KiB coordinate blocks, exact-content duplicates were removed by digest, and all blocks sharing the same source directory were kept on the same train/test side.

Final deduplicated corpus:

- 6,622 blocks/documents
- 46,988,130 bytes = 44.81 MiB
- 5,218 training blocks
- 1,404 held-out blocks
- 921 source groups (host analogue)
- held-out bytes: 6,331,694 = 6.04 MiB

## Operator experiments

### 1. Packed transition keys — REJECTED

Replacing Python tuple keys `(phrase, symbol)` with packed integer keys was expected to reduce allocation overhead. On ~12 MiB of real Python source it was slower:

- tuple-key training: ~1.35 s; frozen encode: ~1.18 s
- packed-key training: ~1.50 s; frozen encode: ~1.35 s

The optimization was reverted.

### 2. Entropy-selected coordinate rotations — REJECTED

Candidate reversible views:

- raw
- XOR with previous byte
- modular delta with previous byte
- XOR with byte two positions back
- modular delta with byte two positions back

A 12 MiB pilot showed that low byte entropy did **not** imply cheap phrase geometry:

- raw phrase MDL: 1.133×
- XOR-1: 0.969×
- delta-1: 0.962×
- XOR-2: 0.885×
- delta-2: 0.876×

Although transformed views won a minority of individual blocks, they destroyed global phrase reuse and made the full rotation corpus dramatically slower. v0.11 therefore requires a cheap phrase-productivity pilot before a transform can be applied at scale.

### 3. Unsupervised multi-vocabulary clustering — REJECTED

Blocks were clustered using only byte-geometry features (16-bin histogram, entropy, zero fraction, printable fraction, high-byte fraction, equality/XOR adjacency). No file-type labels were used. The same 50,000-token total vocabulary budget was divided among eight clusters.

- one global vocabulary: 1.165× held-out MDL
- eight unsupervised vocabularies: 1.084×

Gain = 0.930×. Cross-domain sharing was more valuable than specialization on this corpus.

### 4. Literal-symbol quotient / canonical equality pattern — REJECTED as a blanket transform

Each block was mapped by first occurrence (`ABAB`, `xyxy`, `0101` → `0,1,0,1`) and the reversible symbol map was charged separately.

- raw phrase model: 1.165×
- canonical relation pattern: 0.913×

The transform erased useful literal structure faster than it created transferable relational structure.

## Predictive relation engine

v0.11 adds a second relation family: normalized variable-order byte prediction.

The hierarchy is:

- order 0: `P(x_n)`
- order 1: `P(x_n | x_{n-1})`
- order 2: `P(x_n | x_{n-2}, x_{n-1})`
- order 3: `P(x_n | x_{n-3}, x_{n-2}, x_{n-1})`

The crucial rule is **context-level promotion**. A context either promotes as a whole or does not promote. Promotion can never depend on the symbol currently being predicted. Every promoted context defines a normalized 256-symbol distribution.

### Important bug caught and discarded

An earlier experimental notebook promoted a higher-order relation when the specific `(context,next-symbol)` count passed a threshold. That creates a symbol-dependent model switch and can produce probabilities that do not sum to one. Those earlier ~2× predictive-MDL results were invalid and are explicitly discarded.

v0.11 permanently removes that rule.

## Valid normalized results

### Order 0

- held-out payload: 6.524 bits/byte
- payload ratio: 1.226×
- total MDL: 1.226×

### Order 1

- held-out payload: 5.077 bits/byte
- payload ratio: 1.576×
- actual serialized model: ~151 KiB
- total MDL: 1.517×

### Order 2

Best raw-model MDL at context support >= 4096:

- promoted two-byte contexts: 1,075
- stored children: 144,257
- payload: 4.393 bits/byte
- payload ratio: 1.821×
- serialized model: ~450 KiB
- total MDL: 1.608×
- with zlib-compressed model blob: 1.644×

### Order 3

Best raw-model MDL in the tested grid:

- order-2 context threshold: 16,384
- order-3 context threshold: 1,536
- promoted two-byte contexts: 319
- promoted three-byte contexts: 2,779
- stored order-2 children: 46,416
- stored order-3 children: 150,101
- payload: 4.171 bits/byte
- payload ratio: 1.918×
- actual serialized model: 581,722 bytes
- total MDL: **1.631×**
- with zlib-compressed model blob: **1.675×**

So the valid lift from order 2 to order 3 is small but positive under full model cost.

## Mature-compressor sanity baselines on the exact same held-out blocks

- zlib level 9: **2.346×**
- LZMA preset 6: **2.444×**

RRLS does not beat them here.

That is a useful result: a simple recursive phrase vocabulary plus support-gated Markov relations is not yet a superior general compressor.

## What survives from the theory

The strongest experimentally supported algorithmic rule is now:

1. solve low-order relations first;
2. measure their held-out residual/description cost;
3. allow a higher-order context to exist only if lower-order support justifies it;
4. charge the complete normalized child relation and model serialization;
5. keep the lift only if total held-out MDL decreases;
6. otherwise collapse/back off to the lower relation.

This is a concrete finite version of "dimensional promotion by support".

## v0.11 files

- `rrls_predictive.py` — normalized support-gated predictive analyzer
- `rrls_predictive_cli.py` — run it against any RRLS/Common-Crawl SQLite corpus
- `rrls_commoncrawl.py` / Common Crawl GUI — disk-backed WET sampling and host-disjoint corpus management inherited from v0.10

Example:

```bat
py rrls_predictive_cli.py RRLS_CommonCrawl.sqlite --out predictive_report.json --validation-mib 128
```

## Next falsifiable target

The next promising step is not blindly increasing Markov order. It is to add *different normalized relation families* to the same held-out MDL market:

- word/byte boundary hypotheses
- copy-distance / repeated-substring relations
- periodic/run relations
- numeric delta/recurrence relations
- spatial image transforms
- context-tree / suffix relations

Each operator must compete against RAW, zlib and LZMA, and every learned model must transfer to unseen source groups/crawls without content-digest overlap.