Nothing to read yet
Point any of this section's instruments at a half-trained language model and the most likely reading is: nothing. No dead directions, no orders, an empty result. That is not the instruments failing. All of them share one precondition, and it is about the network, not the tools: a dead direction has to exist before its order can be read. Too obvious to state, except that it is the single most common way a reading comes back empty in practice, and getting the interpretation right matters. A read that returns nothing on a network that has not yet formed singular structure is correct. It has told you what phase of training the network is in.
Two phases
A network's relationship to its capacity changes over training, and the change has two regimes.
In the accumulation phase the network is still absorbing its task. It is recruiting directions, not retiring them: the bottom of the activation spectrum rises as more of the representation comes into use, and no direction is dying anywhere. There is nothing singular about the solution yet, because there is no solution yet, only progress toward one. Most of large-scale pretraining lives here, which is why the reads of this section are not a running commentary on an LLM's training run: they characterise compressed checkpoints, and correctly report the absence of structure on the rest.
In the compression phase the network has the task and starts folding into a singular solution. Spare capacity goes slack, directions begin to die, and the structure this whole section measures comes into existence. Read across training steps, you can watch it happen on a grokking transformer: the dead subspace grows from 188 directions to 562 while its alignment to the coordinate axes climbs from 0.78 to 0.999, and through all of it the order of the deaths holds steady at $k = 3$, the value the architecture's activation fixes.1 The structure sharpens and grows; the integer it carries does not move.
Not everything waits for compression. The gauge#12 directions, the symmetries, are present from initialisation, fixed by the architecture: a LayerNorm model is born with its kernel direction#10. What forms during compression is the learned dead structure, the node-deaths and collapses that carry finite orders. That difference in timing is itself a classifier, a developmental signature that separates the two kinds even before any scan runs.
Train the little network and watch the bottom of its spectrum: rising through accumulation, then turning over and collapsing as compression begins. Press the read button early and it reports, correctly, that there is no structure to read. Press it after the turn and the order comes back. Set the weight decay to zero and the network can sit in accumulation indefinitely, fitting without folding, and the read never fires, which is the correct outcome.
Knowing before you scan
The cheap reads double as the gate for the dear ones. Before running any order scan, three one-pass quantities answer whether a checkpoint has compressed enough to carry anything: the bottom singular value of the activations (still rising means still accumulating), the separation in the K-FAC factors'#6 bottom blocks, and the fraction of directions already near the floor. This is the schedule logic of the cost ladder#11 completing itself: the live reads are not just cheap versions of the expensive ones, they are the trigger that says when the expensive ones have something to measure.
The map of the measurement
With formation in place, the program's instruments sort into three kinds, by what they need and what they return.
Geometry at a checkpoint. Take one frozen, compressed checkpoint and read the structure it carries: each dead direction, its type, its order, the count. Needs no descent and no special optimizer; needs the structure to have formed. This is the read of the next explainer#21, and the rest of this arc.
Formation along the trajectory. Watch the structure emerge during training: the decay-rate reading#4, the developmental signature above, the spectra spreading in the layer-by-layer view#6. Needs the run itself, and, for clean rates, an optimizer that keeps the path legible#18.
The posterior comparator. The established SGLD sampler integrates over the posterior and returns one calibrated global coefficient, at $10^4$ to $10^6$ passes per checkpoint. It is the standard the cheap reads are validated against, and it stays the practical tool where structure resists enumeration (adding up the orders#23).
One precondition under all three: something must have formed. Singular-learning analysis begins where the singularity does.
Still open / underspecified. The two-phase picture is a working boundary, not a theorem: where exactly accumulation hands off to compression, whether a network can compress in one subsystem while accumulating in another (real LLMs almost certainly do), and what the earliest reliable formation signal is, are open. The pre-scan checks answer "has this checkpoint compressed enough to read" conservatively; a finer developmental clock is active work.
- The formation precondition, the developmental signature, and the pre-scan compression checks are from Tejas Pradeep Shirodkar, Measuring Dead Directions: Decomposing and Classifying Singular Structure off Canonical Alignment, arXiv:2607.00603 (2026). ↩︎