Algebraic dead directions
Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale · arXiv:2606.194911
The cheapest read in the program, taken to the largest models. The paper proves that LayerNorm's mean-centring pins one exact dead direction into every model that uses it, the reciprocal scale vector $\gamma^{-1}$, and that RMSNorm, which skips the centring, provably creates none. That yields predictions sharp enough to check to the decimal, and the check runs twice over: at random initialisation, before any training, the predicted direction already matches the measured flattest direction at $|\cos| > 0.9999$ on all nine LayerNorm architectures, and on the pretrained releases themselves the match only tightens, while on five RMSNorm models it is correctly absent, a normalisation classifier that never runs the network. Alongside the kernel sit the residual-stream depth-invariance result, thirteen of fourteen models holding the $\sigma_{\min}$ floor block by block, and the paper's best story: the one exception was not a failed prediction but a genuine dead direction in that model, which the same reads pinpointed to a sub-layer.
It is also the paper of measurement discipline: reference points, precision floors, and calibration distributions, the craft that makes a four-decimal claim at 31 billion parameters meaningful.
Where it is explained here
- The kernel and the classifier: reading a dead direction off the weights#10
- The depth-invariant floor: a floor that survives depth#9
The signature demo
The fourteen-model survey itself:
Of the five, this is the one that carries the geometry to production scale: the same object the theory defines, located by hand inside a 31-billion-parameter model.
- Tejas Pradeep Shirodkar and P. J. Narayanan, Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale, arXiv:2606.19491 (2026). ↩︎