Invariant loss, equivariant optimizer
Slide along a gauge orbit and then take one Adam step, or take the step first and slide after: you land in two different places. Nothing about the loss caused that, since the loss is identical everywhere on the orbit. The update rule did it, and this explainer names the property that separates optimizers that close that square from optimizers that spring it open. A quantity is invariant when the gauge leaves it fixed. A map is equivariant when its output moves in step with its input. The loss is invariant: a gauge move leaves the function, and so the loss, exactly where it was. That much every network has for free, and it is the weaker of the two properties.
An optimizer asks for the stronger property. Its update is a map from a weight setting to the next one, and we want that map to be equivariant: relabel the weights by a gauge move and the update should relabel the same way, so stepping and gauging can be done in either order with the same result.
The demo runs both orders. Slide along the orbit then take a step, or take the step then slide: an equivariant optimizer lands on the same point both ways, closing the square. Switch to the drifting optimizer and the square springs open. The loss was invariant the whole time; that did not save the path.
Why invariance alone is not enough
An invariant loss says the destinations are equivalent. It says nothing about the route. A per-coordinate optimizer like Adam takes a different effective step at different points on one orbit, because its ruler is read off the coordinates the gauge mixes. Two runs that fit equally well drift to different places, and the reading you take at the end depends on where each one happened to park.
Equivariance lets the run descend to the quotient
When the update commutes with the gauge, something clean happens: the whole trajectory projects to a single, well-defined path on the quotient, the space of distinct functions. The vertical, along-the-orbit part of the step projects away, and what is left is pure descent from one function to the next. This is the precise meaning of a gradient flow on the space of functions, and it is the condition that makes the singular-approach rate legible from the path: the run stays on the quotient, so the slope it traces is the geometry's and not the gauge's.
Plain SGD already has this equivariance, for free, under any symmetry that preserves the metric. Adaptive optimizers traded it away for per-coordinate speed. The construction in the next reading buys it back, holding the adaptive preconditioner to the equivariance that SGD never lost.1
Still open / underspecified. Equivariance here is proved gauge family by gauge family, not once for everything a network carries. The scaling and shift gauges close exactly on the Adam base; the rotation gauge closes exactly on an orthogonalising base and to leading order on Adam. A single base optimizer that carries every gauge exactly at once does not yet exist, so "the equivariant optimizer" is really a small kit of constructions, chosen to match the architecture.
- The chain from an equivariant preconditioner to a well-defined flow on the quotient to trajectory-rate readability is the main theorem of Tejas Pradeep Shirodkar, Dead-Direction Conditioners: Gauge-Equivariant Preconditioning for Deep Networks, arXiv:2606.29176 (2026). ↩︎