Why the dead direction depends on the optimizer

The exact symmetries from the start of this section come back here, and this time they are a problem. Scaling a ReLU's weights in against out, rotating an attention head's basis, swapping two matched units: each is a continuous family of weight changes that leaves the computed function exactly the same, what physicists would call a gauge freedom, and real networks are full of them. The softmax is blind to a constant added to all its logits, ReLU layers to a rescaling of weights in against weights out, LayerNorm to a rescaling of its inputs, an attention head to a rotation of its query–key basis. Back in the foundations these were a curiosity, the flat extreme of a dead direction. Here they turn on the instruments: the symmetries decide what a measurement taken in raw weight coordinates can even mean.

Two kinds of gauge: scaling and rotation

These gauge moves come in two shapes, and telling them apart is what makes the rest readable.

Scaling changes magnitudes along fixed axes. The ReLU rescale is this kind: push one weight up by a factor and its partner down by the same factor, and the product, the only thing the unit computes, is unchanged. You met it as a ratio. Which weight is which stays put; only their lengths trade off.

Rotation leaves magnitudes alone and repackages which coordinate holds what. Take two neurons holding values $a$ and $b$. A 45° rotation re-describes them as $(a+b)/\sqrt2$ and $(a-b)/\sqrt2$: the same information, since you can undo it, and the same total size, since $a^2 + b^2$, their combined squared length, is unchanged, but the natural quantities are now a sum and a difference. Scaling stretches along the axes; rotation mixes the axes into sums and differences.

That mixing is what lets a dead direction hide.

When the dead direction lies along a neuron axis, that neuron is simply the dead one, and a check that asks "is each neuron dead?" finds it at once. A rotation tilts the dead direction onto a diagonal, where it becomes a combination of neurons, a sum, with no single neuron dead on its own. The per-neuron check splits the deadness across both and is fooled; only following the direction itself, the Fisher eigenvector, still lands on it. This is the same orbit the demo above traces, now seen as an arrow that should lie on an axis instead lying on a diagonal.

Why rotation shows up in attention but not in a lone MLP neuron. An attention head decides how much each token attends to each other through the dot products between its query vectors $Q$ and key vectors $K$, written $Q K^\top = X\,W_Q W_K^\top\,X^\top$, and nothing else. Replace $W_Q \to W_Q R$ and $W_K \to W_K R$ with $R$ a rotation, so $R R^\top = I$: the rotation cancels in the product and the attention pattern is identical. The head's internal axes are never observed on their own, only through that product, so their orientation can spin freely with no effect on the loss. A ReLU layer is different: it acts on each coordinate separately, so a single neuron's axis is pinned and the only freedom left is to scale it. Rotation reappears only among redundant neurons, a clump that all compute the same dead feature, where any mixture of them is equally dead and something has to choose which combination to call the dead one.

Either way, every point on a gauge orbit is the same function, so a rotation or a rescale only renames the solution you are already at. Fixing the gauge chooses where on the orbit you read or stop; it never rules out a function, because there was never a new function down a gauge direction to rule out.

The function lives in the quotient; a measurement feels the gauge

Picture one note on a guitar, played two ways: the open A string and the fifth fret of the low E string sound the identical pitch. The weights are the fingering; the function the network computes is the sound. The space of distinct sounds, with every fingering that plays the same pitch treated as one point, is the quotient. A dead direction is a property of that space of sounds. Move between two fingerings of one note and nothing you can hear changes, so the function depends only on where you are in the quotient.

But we do not measure in the quotient. We read a dead direction as an eigenvector of the Fisher information, in raw weight coordinates, and those coordinates still carry the gauge. The orbit has a definite tangent at each of its points, and that tangent is the dead direction, but its components in raw coordinates rotate as you slide along the orbit.

The model is identical everywhere on the orbit, yet the dead direction tilts from lying mostly in one weight to mostly in the other. So the question "which weight is the dead one" has no gauge-invariant answer. It depends on where you sit on the orbit, and where you sit is decided by the optimizer: its initialisation, its preconditioner, its weight decay all pull a run to a particular point on the orbit. Two optimizers that fit the data equally well can settle at different points and report different coordinates for the very same dead direction.

Reading through the gauge

This is the missing piece behind the warning a few explainers back, that an optimizer can leave the order unreadable. The order itself is gauge-invariant, but a naive coordinate reading is not, so an optimizer that drifts along the orbit smears the reading out. There are three ways through. You can gauge-fix, choosing a canonical representative of each orbit (the balanced point in the demo, where the in and out weights match) and always reading there. You can measure with an optimizer that is equivariant to the gauge, one whose updates treat every point on an orbit the same, so the run no longer drifts in a way that corrupts the reading; building one is where this arc is headed. Or you can give up on the trajectory entirely and read the order at a single frozen checkpoint, in whatever basis the optimizer left, which turns out to be possible and is the subject of reading at a standstill#21. Every route turns an optimizer-dependent coordinate back into the gauge-invariant order the geometry actually carries.

Still open / underspecified. Which gauge to fix is a genuine choice, and the equivariant route has been built for the four architectural gauge families this section catalogues, with proofs on the scaling gauges and a leading-order construction for the rotations (an optimizer that respects the gauge#17). What stays open is the reach: gauges beyond those families, exactness for the rotation frame, and how far the frozen-checkpoint read substitutes for holding the trajectory clean in the first place.