Along the orbit and across it

Pick a point in weight space and look at the directions leading out of it. They sort into two kinds, and the sorting is the geometric fact the rest of the construction stands on. Some directions run along the orbit: move that way and you slide to another setting that computes the identical function, changing nothing. The rest run across the orbits, from one function to a genuinely different one. The first kind is called vertical, the second horizontal.

Any update you might take is a mix of the two, and the split is an exact projection. The part along the orbit is wasted on the function: it relabels the solution without improving it. The part across the orbit is the genuine step. The demo above takes any direction you choose and separates it, and you can watch the function react only to the horizontal piece.

The quotient is the horizontal world

This is what the quotient is, made local. At each point, the space of distinct functions has exactly the horizontal directions as the ways to move within it; the vertical ones were divided out when the orbit collapsed to a single point. So the natural place for the optimization to live, the space where every direction does real work, is built from the horizontal directions alone.

The two kinds also line up with the program's two faces of a flat direction. The orbit tangent, the vertical direction, is the extreme dead direction, flat to every order because the function never changes along it. The horizontal directions hold the finite-order dead directions, the ones that flatten only as the run approaches a singular point, and those are the ones whose rate the program measures.

An optimizer that conditions in raw coordinates cannot see this split, so it spends part of every step moving along the orbit. Separating the two, conditioning the horizontal part as a real step and handling the vertical part with a single controlled scalar, is what keeps a run on the quotient instead of drifting along the gauge.1

What the split still needs. "Across the orbit" only means something once you say perpendicular measured how: the split needs a metric, and it is exact when the metric is one the gauge preserves. Which invariant metric is the canonical choice, and how the split behaves where orbits curve so the vertical space twists from point to point, are part of what the construction of the later explainers has to handle rather than assume. Where a metric first enters an optimizer, and why that choice is exactly where the gauge bites, comes next#15.

  1. The horizontal and vertical split of the update, conditioned separately, is the core construction in Tejas Pradeep Shirodkar, Dead-Direction Conditioners: Gauge-Equivariant Preconditioning for Deep Networks, arXiv:2606.29176 (2026). ↩︎