A sharper minimum, and a readable rate
Holding a run on the quotient is not bookkeeping. Once the step stops drifting along the orbit, two things change: where the run settles, and how much of the geometry there it leaves you able to read.1
A less degenerate minimum
When the update no longer leaks along the gauge, the run settles into a cleaner, less singular minimum, and the difference shows up on real models. A language model trained well past the point of fit holds a validation-train loss gap of $0.67$ under the gauge-equivariant optimizer, where a matched AdamW blows out to $5.88$: the drifting run keeps sliding into the over-trained collapse, and holding the quotient resists it. A vision transformer trained from scratch reaches lower validation loss, $1.71$ against $2.12$, and compresses the spare feed-forward capacity that the matched AdamW leaves sitting there under the same weight decay. The minimum is also tidier in a way you can score: measure how nearly each dead direction lies along a single coordinate axis, with $1$ meaning perfectly axis-aligned, and the equivariant run's structure scores $0.889$ against the matched baseline's $0.718$ on the grokking transformer, deadness concentrated in nameable coordinates rather than smeared across them. Its smallest Fisher eigenvalue lands more than ten times higher too, so the optimizer has parked further from the singular set it was sliding toward.
On a Muon base, where the rotation gauge composes exactly, the steadiness is sharpest: at depth 24 the gauge-equivariant run groks on ten of eleven random seeds, where a plain Muon groks on none.
A rate you can read
The second change is the one the whole program is built toward. Because the trajectory stays on the quotient, the dead-direction rate is legible straight off the path. The smallest Fisher eigenvalue traces a clean power law as the run approaches the singular set, and its log–log slope, $2(k-1)$, hands you the order $k$. A drifting optimizer contaminates that reading with a gauge-leak floor that does not vanish, so the same geometry reads as a curve that flattens before it arrives and a slope fit that lands on the wrong number.
The gap is large in practice. Scanning a language model layer by layer, with several spectral observables read at each layer, gives a grid of 65 places a rate could in principle be read. The gauge-equivariant optimizer leaves a clean, fittable power law in 32 of them; AdamW leaves one in 7. Holding the run on the quotient turns the optimizer into a measurement instrument for the singular geometry: the run it takes to fit the model is also the run that measures how degenerate the minimum it found is.
What it does not buy
Respecting the gauge is not a free speed-up, and the accounting matters. On the grokking transformer the speed itself comes mostly from the weight decay that travels with the projection: hold the weight decay equal on a shallow, tuned grid and the plain baseline groks cell for cell alongside the gauge-equivariant run. What the projection earns instead is reliability where the problem gets hard, and the depth-24 result above is exactly that, ten of eleven seeds against none at the same weight decay. It also leaves the cleaner axis-aligned structure, and it keeps the rate legible. The exact equivariance is proved for the scaling and shift gauges on the Adam base; the rotation gauge closes exactly on the Muon composition and to leading order on Adam, so the reach is bounded and stated as such.
So the trade lands like this: a sharper minimum, a steadier path to it, and a geometry that is still readable when you arrive. A network's loss is blind to its own symmetries, and an ordinary optimizer inherits that blindness as a drift it cannot resolve. Building the symmetry into the preconditioner removes the drift, and the gauge an optimizer respects ends up deciding both the minimum it reaches and how much of that minimum's geometry it leaves you able to measure. And if a run was not trained this way, it is not lost to measurement: the trajectory read here is the descent-time instrument, and the frozen-checkpoint read#21 recovers the same order after the fact, in whatever basis the optimizer left behind.
Still open / underspecified. The numbers carry their own caveats. Ten of eleven seeds is not eleven: one seed still fails to grok, and why is open. Thirty-two of sixty-five readable cells is not sixty-five: which layers and observables stay unreadable even on the quotient, and whether that is finite-run noise or real structure, is part of the measurement work. And the reach is bounded to the four architectural gauge families, on the bases that carry them.
- The over-training, vision-transformer, depth, axis-alignment, and rate-readability results, and the weight-decay attribution behind that accounting, are from Tejas Pradeep Shirodkar, Dead-Direction Conditioners: Gauge-Equivariant Preconditioning for Deep Networks, arXiv:2606.29176 (2026). ↩︎