Quantisation breaks the LayerNorm gauge

Given the activations a transformer produces on its own forward passes, which of its weights are determined? The answer splits the parameters into three populations of genuinely different status. Maps not fed by a LayerNorm are determined outright by least squares. Maps fed by a LayerNorm are not determined: the normaliser's centring induces an exact one-dimensional gauge symmetry, so an entire line of weight vectors produces identical outputs, and least squares provably returns the wrong point on that line. The embedding tables carry a second gauge the first argument cannot touch.

The obstruction is real in ℝ. It is not real in float32. A gauge orbit is a continuous line; the stored weights lie on a discrete grid; and a line through a 769-dimensional lattice passes within rounding distance of a grid point in every coordinate at one place. Enumerating 13,645,404 candidates for a single column yields exactly one survivor with all 769 coordinates bit-exact. The probability a wrong candidate survives is 10⁻⁶²⁵⁰ under the null stated in §4.

The instrument sees activations, never weights

Run N token sequences through the model. At each linear map record the input activations X (N × fan_in) and the output activations Y (N × fan_out). No parameter is ever read. The question is whether X and Y determine it.

Unless stated otherwise, N = 8,192 (8 sequences of 1,024 tokens), seed 32, with the token ids hashed to 50c674a8d7a91b94… so a replication uses the same probes.

Full-rank designs are determined outright

For a map whose input is not a LayerNorm output — in GPT-2, attn.c_proj and mlp.c_proj — write Y = XW + b and augment with a column of ones, Xa = [X, 1].

If rank(Xa) = fan_in + 1 this is the unique solution: no other pair reproduces Y. That needs N > fan_in + 1, which is why the observation count may never drop below the largest fan-in plus two, 3,074 here. Solve in extended precision, then round to the nearest float32; rounding is safe because the residual sits fourteen orders of magnitude below half an ulp.

Across the model this population is 35,407,872 parameters (35,446,272 with the LayerNorm gains and biases, which are also closed by least squares). It is not the interesting part.

LayerNorm hands its consumers an exact symmetry

LayerNorm emits ln_j = g_j·x̂_j + β_j where x̂ has been centred, so ∑_j x̂_j = 0. Divide by g_j and sum over the feature index:

The left side is built from observations. The right side is a constant of the layer. The equality holds for every input the network will ever see, so the augmented design is rank-deficient in a direction known in closed form:

This is not a numerical near-degeneracy. It is a symmetry of the function being observed, and it has a blunt consequence: a one-parameter family of weight matrices is observationally indistinguishable. Least squares must pick one, and it picks the minimum-norm point — the one orthogonal to v — which is not the point the checkpoint stores.

Rank-deficient by exactly one, every time. The null direction sits at cosine 1.000000 to the predicted (1/g, −C), so the deficiency is the derived symmetry and not numerical noise. The fit reproduces outputs to within 1e−13 while almost no parameter values match bit for bit. A functional copy with different stored weights is exactly what Theorem 1 predicts.

A line through a lattice lands once

The corollary is a statement about ℝ. The weights do not live in ℝ. They live in float32, a discrete set, and the gauge orbit is a line drawn through it:

Almost every point on that line has coordinates that are not float32 numbers. Recovery becomes a question of simultaneity: for which α do all 769 coordinates land on grid points at once?

Move α by more than min_j(½ulp(W_j)·|g_j|) and at least one coordinate falls off its grid point. For column 0 of h.0.attn.c_attn that tolerance is 3.204e−12 against a nominal search window of 0.25 (0.074 for this column after the per-column cap). Score a candidate by its worst rounding residual across the 769 coordinates, in ulp.

The 3.725e−09 score is measured at the checkpoint's stored α; the survivors found from observations score between 2.0e−05 and 8.65e−04. The probability in Theorem 2 uses the stored-α score.

The survivor is not the best of a field. It is the only member of the field, by a margin with no physical analogue. And the enumeration was run:

The embeddings need a different argument entirely

The token and position tables carry their own additive gauge, and §4 does not break it. The reason is exact rather than practical: a uniform shift of a feature is itself a lattice translation. It carries grid points to grid points, so the float32 grid supplies no constraint at all. The lattice attack was run anyway and reached 820,496 of 39,383,808 values, 2.08% — a negative result reported because it delimits the method.

What closed the embeddings avoids the gauge instead of breaking it. The tied readout observes wte without wpe, and 768 logit equations fix the gauge constants outright: 39,383,808 of 39,383,808, with a shuffled control at 0 of 39,383,808.

Eight weights that needed more evidence, not better arithmetic

Eight parameters sit where the float64 oracle's own rounding is the noise floor, so at N = 8,192 their last bit is a coin toss. No amount of precision fixes that; more observations do, because the decision margin grows as √N. A second stage draws fresh sequences for those columns alone until every margin reaches 4: on the first machine, 8 stragglers, 27 chunks, margins 4.012 to 16.816, at 221,184 observations.

The hardest is h.3.mlp.c_fc column 2350. Independently of any of this, the only non-positive LayerNorm gain in the entire checkpoint is h.3.ln_2.weight[266] = −2.5572e−04, whose 1/g = −3,910.5 ranks first of 768 by magnitude in that block, 1,218× the median. The worst-conditioned coordinate in the model and the last weight to close are the same coordinate. That correspondence was available to fail and did not.

Every parameter, on two unrelated machines

Misses: none. The assembled file's sha256 is 248dfc3911869ec493c76e65bf2fcf7f615828b0254c12b473182f0f81d3a707, equal to the published checkpoint's.

Independence

The run was repeated on a machine sharing no instruction set, operating system or extended-precision implementation with the first.

The two machines took measurably different routes — 88 columns needed the extended-precision path on one, 89 on the other — and produced identical bits. The path is contingent; the answer is not.

A method that always succeeds has shown nothing

Permute the rows of X, breaking their pairing with Y, and run the identical solve. If the recovered bits came from anything other than the observations, this would change nothing.

No parameter values matched bit for bit, with residuals fifteen orders of magnitude larger. The result depends on the observations and on their pairing.

Why this paper publishes its method

pip install numpy huggingface_hub
hf download openai-community/gpt2 model.safetensors --local-dir gpt2

# the gauge algebra, against the public checkpoint, in seconds
python src/check_gauge.py gpt2/model.safetensors

# one block, the lattice break, and the shuffled-X falsifier
python src/gpt2_demo.py gpt2 --blocks 1 --pin-cols 8

# all 124,439,808 parameters, assembled and hashed
python src/gpt2_full.py gpt2 --out full.json

The mathematics above is the method. §2 is least squares; §3 is two lines of algebra anyone can verify against the checkpoint; §4 is an enumeration whose statistic is calibrated against its own null in Figure 3. What remains unpublished is unrelated work (a larger-scale pipeline), and none of it is load-bearing here.

This paper has not been peer reviewed. Before publication the argument was put to two language models in separate sessions and prompted to attack it; that is adversarial review, not peer review, and it is reported as such. They pressed the copying objection above, which is why §8 exists at all, and they found two defects in the earlier write-up: a mislabelled comparison column, and a one-sided grid tolerance that returned negative half-widths against the model's single negative LayerNorm gain. Both are corrected here. The mathematics of §3 and §4 came through unamended apart from that tolerance fix, which is weaker evidence than a referee's signature and is the only such evidence on offer.