The symmetries are groups
Rescale a ReLU unit's weights, then rescale them again: the two moves collapse into a single rescale, by the product of the two factors. Undo one with its reciprocal and you are back exactly where you began. Even doing nothing at all counts as a move, the trivial one. A family of moves closed in this way, where composing stays in the family, everything can be undone, and doing nothing is included, is what mathematics calls a group, and every gauge freedom you have met is one: the logit shifts, the rescalings, the head rotations. This is worth caring about because the network only ever feels its weights up to the moves of these groups, so the group is the exact shape of the redundancy, and a single property of each group decides how hard its redundancy is to handle.
When the order of two moves matters
A group is abelian when the order of two moves makes no difference, and non-abelian when it does. Stretching one axis then squashing another lands exactly where squashing then stretching does. Rotating a shape then flipping it does not match flipping then rotating.
The distinction is not a curiosity. An abelian gauge is, in effect, a set of independent dials, one number per direction, and a single number records where on the orbit you sit. A non-abelian gauge has no such coordinate: there is no one number that says "which rotation," so pinning your place on the orbit takes a whole frame, a full set of axes.
The four gauges, sorted
The architectural gauges fall into the two classes cleanly.
Three are abelian. The cross-entropy shift adds the same constant to every logit, a translation. The ReLU and SwiGLU rescaling multiplies a weight in by $c$ and the partner weight out by $1/c$, a positive scaling. The LayerNorm and RMSNorm scale does the same per channel. Each is one number per direction, and the moves commute.
One is non-abelian. The attention head rotation turns a head's query and key basis by any orthogonal matrix, the group $O(d_{\text{head}})$, and these compose in an order-dependent way. This is the rotation you met inside attention, where only the product of the query and key weights is ever observed, so their shared basis can turn freely.
That split is the reason the handling differs, and it sets up everything the next explainers build. For an abelian gauge, one number per channel is a complete record of where you sit on the orbit, so an optimizer can account for the whole symmetry with a scalar. For the head rotation, no scalar can stand in for a rotation, and the optimizer has to carry a whole frame, the head's own axes.1 Before an optimizer can use any of this, though, there is a piece of geometry to set up: at every point in weight space, which directions belong to the symmetry and which do real work. Drawing that line is the next explainer#14's whole job.
Still open / underspecified. The inventory here covers the continuous architectural gauges; the discrete ones, like permuting identical units, need different handling and are not part of this construction. And even within the continuous ones, matching the gauge to the symmetry the architecture actually carries is a real decision: a rotary-embedding attention head is symmetric under a torus of paired plane rotations, not the full rotation group, and building for the torus rather than the generic group measurably improves the optimizer built on it (final accuracy 0.977 against 0.955 on the grokking testbed, with half the seed-to-seed variance). The discrete symmetries, and the per-architecture work of finding the gauge a network actually carries, are the open edges.
- The four-gauge inventory, the abelian-versus-rotation handling, and the torus-versus-generic rotation comparison are from Tejas Pradeep Shirodkar, Dead-Direction Conditioners: Gauge-Equivariant Preconditioning for Deep Networks, arXiv:2606.29176 (2026). ↩︎