How an optimizer measures distance
Gradient descent carries a hidden choice. "Move downhill" is not a complete instruction until you say downhill measured how. The plain gradient step $-g$ is the steepest-descent direction only when distance is measured the round, Euclidean way. Measure it differently and the steepest direction itself moves, even though the gradient has not.
A way of measuring distance is a metric, and an optimizer that rescales the gradient is choosing one. With metric $M$ the step becomes $-M^{-1} g$: the gradient, re-measured. The metric's unit ball is the ruler, and the step is steepest descent against that ruler.
The gradient at the point never moves in the demo. Only the metric does, and the step swings with it. Tilt the ruler to match the valley and the step turns to point straight at the bottom.
Three optimizers, three rulers
The familiar optimizers are three choices of metric. Plain SGD uses the round one, $M = I$, so its step is just $-g$. Adam uses a per-coordinate ruler, an axis-aligned ellipse whose stretch along each axis comes from that coordinate's recent gradient size. The natural gradient uses the Fisher information, the same curvature of the model's own predictions met earlier as how sharply the loss bends. That ruler matches the geometry, so its step points much closer to the true bottom of the valley than the raw gradient does.
Why this is the lens for the gauge
Seeing the optimizer as a choice of metric is what turns "respect the symmetry" into something precise. Conditioning in a metric the gauge leaves alone means choosing a ruler the gauge cannot distort. Adam's per-coordinate ruler is distorted by the gauge: a rotation mixes the very coordinates the ruler treats one at a time, so the same physical step gets measured differently depending on where the run sits on the orbit. A ruler built to be invariant measures the same step the same way everywhere on the orbit.
That is the move the construction makes: replace Adam's per-coordinate ruler with one taken in the orbit decomposition of a gauge-invariant metric, close in spirit to the Fisher, so the conditioning means the same thing at every point on the orbit.1
Still open / underspecified. There is an irony to sit with here. The geometrically ideal ruler, the Fisher itself, is exactly the thing that degenerates at the singular set: the natural gradient's metric blows up precisely where this program points its instruments. Every practical invariant metric is therefore an approximation, kept well-conditioned on purpose, and how faithfully a given choice preserves the rate you are trying to read is part of the measurement work rather than a solved preliminary.
- The gauge-invariant preconditioning metric and its orbit decomposition are from Tejas Pradeep Shirodkar, Dead-Direction Conditioners: Gauge-Equivariant Preconditioning for Deep Networks, arXiv:2606.29176 (2026). ↩︎