Dead-direction conditioners
Dead-Direction Conditioners: Gauge-Equivariant Preconditioning for Deep Networks · arXiv:2606.291761
The theory paper's quotient theorem says which optimizers leave the singular geometry legible; this paper builds one. DDC decomposes every update against the network's own symmetries, the cross-entropy shift, the ReLU and SwiGLU rescalings, the normalisation scales, and the per-head attention rotations, conditioning the genuine descent with full adaptivity and the symmetry component with a single controlled scalar, then re-projecting so the step lands on the quotient. Equivariance is proved exactly for the scaling gauges on the Adam base and carried natively by an orthogonalising base for the rotations. The headline results are what holding the quotient buys: an over-trained language model's validation-train gap held at $0.67$ against a matched AdamW's $5.88$, a from-scratch ViT reaching $1.71$ against $2.12$, grokking on ten of eleven seeds at depth 24 where plain Muon at the same weight decay groks on none, and a rate readable in 32 of 65 layer-observable cells against AdamW's 7. The accounting is part of the result: the grokking speed belongs to weight decay; the projection buys reliability and a reading.
Where it is explained here
- The problem, in one demo: why the dead direction depends on the optimizer#12
- The build-up: the symmetries are groups#13 → along the orbit and across it#14 → how an optimizer measures distance#15 → invariant loss, equivariant optimizer#16
- The construction: an optimizer that respects the gauge#17
- The payoffs and the weight-decay accounting: a sharper minimum, and a readable rate#18, why grokking is slow then sudden#19
The signature demo
Eleven seeds a side at depth 24, weight decay matched:
Here the program turns operational: the geometry stops being something you measure about training and becomes something you train with.
- Tejas Pradeep Shirodkar, Dead-Direction Conditioners: Gauge-Equivariant Preconditioning for Deep Networks, arXiv:2606.29176 (2026). ↩︎