Glossary

Each term below is defined where it is taught; this glossary is the one-stop version. The link on each entry goes to the explainer that owns the idea.

Dead direction: a direction in weight space along which the model's sensitivity dies: the Fisher information along it collapses as the singular point is approached, and the loss flattens with a measurable order. Exact symmetries are the extreme, flat-to-every-order case. Owned by what is a dead direction#3.

KL order $k$: the integer power with which the KL divergence to the truth climbs, $K \sim t^{2k}$, as you step distance $t$ along a direction. Regular directions have $k = 1$; dead directions $k \ge 2$. Owned by what is a dead direction#3.

Fisher information / Fisher metric: the matrix that scores how sharply the model's predictions move per unit step in each direction; it degenerates along dead directions, decaying as $t^{2(k-1)}$ on approach. Owned by reading the order#4.

Singular set: the set of parameters that exactly achieve the best fit; a surface rather than a point in an overparameterised model, folded and crossed by gauge orbits. Owned by what is a dead direction#3.

Real log canonical threshold (RLCT), $\lambda$: a singular model's effective complexity. Per direction, $\lambda_{\mathrm{dir}} = 1/(2k)$; globally, the assembled coefficient that prices the model in Watanabe's free energy and sets the generalisation error $\lambda/n$. Owned by why singular models generalize#5 and adding up the orders#23.

Multiplicity $m$: the number of dead directions attaining the flattest order at a singular point. On trained networks the measurable analogue is the dead-subspace dimension, equal to $m$ only when the dominant directions form a single crossing. Owned by counting dead directions#8.

Singular fluctuation $\nu$: the asymptotic posterior variance of the log-likelihood; sets the train–test gap $2\nu/n$ and the WAIC correction. Universal in the order per isolated direction, partly absorbed by live structure on trained networks. Owned by why singular models generalize#5 and adding up the orders#23.

K-FAC factorisation: the per-layer approximation $F_W \approx A \otimes G$: a layer's Fisher as the Kronecker product of its input covariance $A$ (forward) and output-gradient covariance $G$ (backward). Owned by reading the order one layer at a time#6.

A–G duality: a dead direction announces itself in the forward factor, the backward factor, or both; reading whichever side separates it more cleanly is both a detector and a cross-check. Owned by reading a dead direction from both sides#7.

$\sigma_{\min}$: the smallest singular value of a layer's activations: the flattest direction of the representation, one forward pass, held comparable across depth by the residual stream. Owned by a floor that survives depth#9.

Rank-deficit $r$: the number of directions collapsing together at a layer, counted by the slope of the active volume $\log\det^{+} G$, which falls $r$ times as fast as a single direction. Owned by counting dead directions#8.

LN-kernel direction: the reciprocal of a LayerNorm's scale vector, $\gamma^{-1}$, normalised: an exact algebraic dead direction created by mean-centring, readable off the weights with no forward pass, and absent in RMSNorm. Owned by reading a dead direction off the weights#10.

Gauge: a continuous family of weight changes that leaves the network's function exactly unchanged: the ReLU rescale, the LayerNorm scale, the cross-entropy shift, the attention-head rotation. Gauge directions are flat by symmetry, carry no order, and are booked apart from genuine deaths. Owned by why the dead direction depends on the optimizer#12.

Orbit: the path a gauge sweeps through weight space; every point on it computes the same function. Owned by what is a dead direction#3.

Quotient: the space of distinct functions, with each orbit collapsed to a point; the place where dead directions actually live and where an equivariant optimizer's trajectory descends. Owned by along the orbit and across it#14.

Vertical / horizontal: the split of directions at a point into along-the-orbit (pure gauge, changes nothing) and across-orbits (real functional change). Owned by along the orbit and across it#14.

Invariant / equivariant: a quantity is invariant when the gauge leaves it fixed; a map is equivariant when it commutes with the gauge. The loss is invariant for free; an optimizer must be equivariant for its run to project to a clean path on the quotient. Owned by invariant loss, equivariant optimizer#16.

DDC (dead-direction conditioners): the gauge-equivariant preconditioning construction: split each update into vertical and horizontal parts against an invariant metric, condition each on its own terms, re-project. Lifts Adam and Muon alike. Owned by an optimizer that respects the gauge#17.

Grokking: a late, sudden jump from memorisation to generalisation after a long flat plateau; on the flat-landscape account, the plateau is weight decay's slow walk to the low-norm exit. Owned by why grokking is slow then sudden#19.

Accumulation / compression: the two phases of training: recruiting capacity (bottom of the spectrum rising, no singular structure) versus folding into a singular solution (dead directions forming). Every read requires the second. Owned by nothing to read yet#20.

Frozen read / off-canonical read: the descent-free, alignment-free order measurement: construct the dead direction from the K-FAC factors at a single checkpoint, scan a synthetic displacement, fit the purity-matched window. Owned by reading at a standstill#21.

Node-death: a hidden unit whose incoming and outgoing weights have both collapsed; a genuine singularity whose order is the activation's analytic order. Owned by every way a direction dies#22.

Magnitude criterion: the tie-breaker between a curved gauge orbit (slope 2 at a floor orders of magnitude below live) and a genuine $k=2$ death (slope 2 rising to live values). Owned by every way a direction dies#22.

Sum-versus-crossing rule: how per-direction coefficients assemble: independent loci add their $1/(2k)$; crossing loci take the minimum, with $m$ counting the directions that attain it. Owned by adding up the orders#23.

WAIC: the Widely Applicable Information Criterion: training loss plus the posterior's functional variance, estimating test loss even for singular models. Owned by why singular models generalize#5.