A modern neural network has far more parameters than its task strictly needs, and training quietly takes advantage of that. Watch a network train and something happens alongside the fitting: whole directions in weight space fall out of use. The model's sensitivity along them dies away, step by step, until moving the weights that way barely changes anything the network does. Such a direction is a dead direction, dead because the sensitivity along it has died.
Where dead directions gather, the geometry of the loss caves in. Many nearby settings of the weights now produce almost the same predictions, so the best fit is no longer a point but a whole flat region, and the model sits at what mathematics calls a singularity. Dead directions are the paths training takes into these places, and the flat directions it finds once there.
There is a version of this every network is born with. Double all the weights going into a ReLU unit and halve the weights coming out, and the network computes literally the same function; spin an attention head's internal axes, and it attends to exactly the same tokens. Along these built-in symmetries nothing was ever alive to die: the direction is perfectly flat from the start, and there is nothing to measure. The dead directions worth studying are different. They die by degree: the function still changes along them, only ever more slowly, and how fast it flattens is a number you can measure. That number, what it tells you about the model, and how to read it from a real network, is what this section is about. Telling a symmetry apart from a genuinely dying degree of freedom turns out to be part of the job.
None of it happens all at once, either. The slide toward a singularity is a phase of training, not a permanent condition: a network still absorbing its task, which is where most large-scale pretraining lives, has not yet formed the structure, and knowing when there is something to read is part of the work too.
The two theories it connects
Two mathematical traditions have circled this degeneracy from opposite sides.
Information geometry treats a family of models as a smoothly curved surface, with the Fisher information setting distances on it. It is sharp and powerful in the well-conditioned regime, and it loses its footing at exactly the singular points, where the surface stops being smooth.
Singular learning theory, developed by Sumio Watanabe, was built for those points. It is the theory of the singular regime, the place information geometry cannot reach with its usual machinery.
Historically the two have stayed apart, using different mathematics and describing opposite regimes. The dead direction is the object that sits between them, on the edge where the well-conditioned world hands off to the singular one.
The goal
The aim of this research is to make that connection precise, and above all measurable. A single dead direction can be read from either side: as the geometry of a singular loss valley, the way Watanabe's theory sees it, and as the collapse of the Fisher information, the way Amari's geometry sees it. Read either way, how fast it flattens comes out as the same number. Everything here grows from that one fact, because it means a quantity from the deep theory, the kind that once took algebraic geometry to extract, can be read off an ordinary training run.
From there the work follows the connection outward: what the geometry implies for how networks generalise and grow their structure, how it can be measured at the scale of real models, and what it unlocks for the way we understand and build artificial intelligence.
The program so far
The program has produced five papers to date. The first lays the theoretical ground, proving that the flatness of a dead direction, one integer, can be read from the slope of a decay curve. The four after it each put a different instrument on the same object: a layer's spectrum, which detects dead directions and counts how many are collapsing together; a normalisation layer's weight vector, which carries one in closed form before the model has computed anything; the path a symmetry-respecting optimizer traces, which keeps the order legible during training; and, most recently, a single frozen checkpoint, from which the order can be recovered with no training run in sight. Wherever two instruments read the same quantity they agree, and that agreement is what convinces us the structure is a property of the model rather than of any one instrument. The thread is still being pulled: each paper closes one question and opens others, and the open ends are marked throughout.
The papers are written for reviewers. This section is written for everyone else. We have built it as a set of explainers with interactive demos, because the geometry at the heart of the program is genuinely visual and most of its ideas can be operated: every explainer introduces one idea in plain language and then hands you the controls, so you can watch a direction die, read its order off a slope yourself, and break the assumptions to see what fails. Foundations builds the core picture, from a single neuron to why flatness decides generalisation. Reading & Measuring is the instrument shop, the same direction read from the forward side, the backward side, and the weights themselves, on a ladder of cost. Gauge & Optimizers faces the complication: the network's own symmetries make the reading depend on the optimizer, and an optimizer built to respect them turns training itself into the measurement. The Full Reading takes any trained checkpoint where the structure has formed and reads it whole. The Papers maps each arXiv paper to the explainers that teach it.
Foundations
What is a neuronA network is built from one small part repeated: a neuron that weighs its inputs, sums them, and shapes the total into an output. Operate one by hand, scale it up to an image, stack a few into a network, and the central surprise appears: the same output can be reached by many different weights, so the best fit is a whole region of weight space.
Why neural networks are singularThe textbook says an overparameterised network should memorise and fail; it generalises instead. The resolution is geometric. The map from weights to functions is massively many-to-one, so the best fits form a folded, crossing surface, the loss landscape is singular, and the classical toolkit was never valid there to begin with.
What is a dead directionChange a network's weights without changing what it computes. A ReLU unit's in and out weights trade off exactly, and the direction they trade along is your first dead direction, the perfectly flat extreme. The general case asks for less, a sensitivity that dies by degree, and the degree is an integer you can measure.
Reading the order of a dead directionNot every dead direction is equally dead. The difference is one integer, the order k, and it can be read two ways that have no right to agree, from how the KL divergence vanishes (Watanabe) and from the slope of the Fisher decay (Amari). Slide both and watch them lock.
Why singular models generalizeThe opening puzzle, resolved. A flatter valley holds more near-optimal volume, so a singular model's effective complexity λ falls below its raw parameter count, and that same λ sets the generalisation error λ/n. Slide the order and watch geometry and generalisation move together.
Reading & Measuring
Reading the order one layer at a timeA transformer's full Fisher is astronomically large. But per layer it factorises as A ⊗ G, forward activations times backward gradients, so every eigenvalue is a product and a dead direction is just one slack factor. Pull either down and a whole row or column of the layer dies.
Reading a dead direction from both sidesThe bridge made measurable. A dead direction can be read from the activation side (σ_min) or the Fisher side (λ_min(G)); when both sides see the same death the reads lock onto a line of slope 2, and when they part ways the gap itself says where the death lives. Geometric, so it survives regimes where the decay-rate reading breaks.
Counting dead directionsKnowing a dead direction is there is not the same as knowing how many. The smallest eigenvalue is rank-blind, its slope is 2 whether one direction is dying or ten. The volume of the Fisher, log det⁺ G, adds up every live eigenvalue, so its slope is 2r and divides out the count.
A floor that survives depthStack enough layers and the weakest direction in the activations gets squeezed to nothing. A residual stream's identity path stops the compounding, so σ_min holds steady with depth and reads the same at any layer. Toggle the skip connection and watch the floor die.
Reading a dead direction off the weightsFor a LayerNorm, one dead direction needs no forward pass. Centring the input starves the all-ones direction of variance; the γ scale tilts it to γ⁻¹, which you read straight off the weights. RMSNorm skips the centring and has no such direction, so the kernel even tells you which normalisation a model uses.
The cost of reading a dead directionThe same dead direction can be read at four prices, from a glance at a weight vector to a full Fisher spectrum, with the posterior sampler a thousand times dearer still. Each price buys an answer to a different question, presence, location, count, so the rule is to pay for the question you are actually asking. Even the ordering moves; the spectrum overtakes the cheap reads as width grows.
Gauge & Optimizers
Why the dead direction depends on the optimizerThe weight-space symmetries are gauge freedoms, continuous moves that leave the function fixed. A dead direction lives in the quotient, but we measure it in raw coordinates, so its components depend on where the optimizer parked on the orbit. Slide along the orbit and watch the reading tilt while the model holds still.
The symmetries are groupsRescale a ReLU's weights twice and you have rescaled once, by the product; undo with the reciprocal; doing nothing counts too. Each gauge freedom is a group, and one property, whether the order of two moves matters, decides everything downstream. Three of the four architectural gauges commute and pin down with a single number; the attention rotation does not, and needs a whole frame.
Along the orbit and across itStand at a point in weight space and the directions you could move split into two kinds: vertical ones along the orbit, pure gauge that change nothing, and horizontal ones across orbits that actually change the function. The quotient is the horizontal world, and an optimizer that mixes the two is what drifts.
How an optimizer measures distanceGradient descent hides a choice: go downhill measured how. A step is steepest descent only in some metric, and changing the metric swings the step. SGD picks the round one, Adam a per-coordinate one, the natural gradient the Fisher. Seeing an optimizer as a choice of metric is what makes conditioning in a gauge-invariant metric mean something.
Invariant loss, equivariant optimizerTwo words that sound alike and pull apart under load. The loss is invariant under a gauge, unchanged by it. What a symmetry-aware optimizer needs is something stronger: an equivariant update, one that commutes with the gauge. Only then does the whole run project to a single clean path on the space of distinct functions.
An optimizer that respects the gaugeA network's loss is blind to its own symmetries, and an adaptive optimizer inherits that blindness as a drift along the orbit it cannot see. Adam's per-coordinate step leaks along the gauge. DDC splits every update into the descent across orbits and the flat freedom along them, conditions each in a frame that turns with the symmetry, and holds the run on the quotient.
A sharper minimum, and a readable rateHolding a run on the quotient changes where it ends up and what you can measure once it gets there. The minimum is less degenerate, more axis-aligned, and steadier across seeds, and the dead-direction rate is legible straight off the path. The fine print: the speed itself is the weight decay, and what respecting the gauge earns is reliability and a reading.
Why grokking is slow then suddenA network can sit at low training loss for ages and then suddenly generalise. The flat-landscape picture explains it: a true symmetry plateau has no door to find, but a real memorising plateau does, and weight decay is what walks the weights to the low-norm exit. Slow for a long time, then a sudden drop.
The Full Reading
Nothing to read yetEvery read in this program shares one precondition, a dead direction has to have formed. A network still absorbing its task carries no singular structure, and a read that returns nothing there is a correct answer about the network's phase, not a failure. Watch a network cross from accumulation into compression and the moment the reading switches on.
Reading at a standstillThe trajectory reading needed a descending run and coordinates aligned with the geometry. Drop both. At a single frozen checkpoint, construct the dead direction from the layer's two K-FAC factors, walk a synthetic path along it, and the directional Fisher hands back the order, in whatever basis the optimizer left. Rotate the structure off the axes in the demo and watch the axis-by-axis scan fail where the constructed read holds.
Every way a direction diesOne read, many kinds of structure. A finite order marks a genuine singularity whose value the architecture fixes, a node-death at the activation's order, a deep-linear collapse at the depth. A flat scan at a deep floor marks a gauge. The subtle case is the curved gauge orbit that imitates slope 2, told apart only by magnitude, and the headline is that the order is an invariant: optimizers choose the basis, the architecture chooses the integer.
Adding up the ordersThe reads return one order per direction; the theory wants one λ per model. The assembly rule is sum versus crossing, independent directions add their 1/(2k), intersecting ones take the minimum, and it reproduces the closed form to machine precision on every enumerable cell. The hard boundary is where enumeration fails, and ν brings the last surprise, fixed by the order in theory, partly absorbed by the live network in practice.
The Papers
Geometric singular learningThe hub of the program. Defines the dead direction as the object shared by Watanabe's singular learning theory and Amari's information geometry, proves that one integer, the KL order, is readable from a Fisher decay slope, and builds the bridge out to deep networks and optimizers.
Dead-direction signaturesThe spectral spoke. Three cheap observables, the activation floor, the smallest Fisher eigenvalue, and the active volume, detect, localise, and count dead directions from single passes, with the volume identity as the discriminating test single-eigenvalue monitors cannot run.
Algebraic dead directionsThe algebraic spoke. A LayerNorm's mean-centring creates one exact dead direction, γ⁻¹, readable from the parameters alone with no forward pass; RMSNorm provably has none. Verified to four decimal places on nine LayerNorm models and correctly null on five RMSNorm ones, up to 31 billion parameters.
Dead-direction conditionersThe optimizer spoke. DDC splits every update into the part that descends and the part that only slides along a symmetry, conditions each on its own terms, and holds the run on the quotient. The payoff is a cleaner minimum, seed-level reliability at depth, and a training path you can read the geometry from.
Measuring dead directionsThe methodology spoke, and the newest. Drop the descent and the alignment preconditions entirely, read each dead direction's order at a single frozen checkpoint in whatever basis the optimizer left, classify what kind of structure it is, and assemble the per-direction orders into the model's global complexity.
Reference
All the demosEvery interactive demo on the site, in reading order, with the explainer and section it lives in. Each one is a claim you can take apart with your own hands.
GlossaryThe section's working vocabulary in one place, each term with its canonical definition and a link to the explainer that owns it.
