Why grokking is slow then sudden
A network often fits its training data fast, holds full training accuracy while validation stays near zero, and only much later does validation suddenly jump. This is grokking,1 and it looks mysterious: nothing seems to move for thousands of steps, then validation climbs off the floor. The flat-landscape picture from the rest of this section explains it, and the explanation starts with a thought experiment.
Picture a flat region of the loss with one much deeper minimum reachable only through a single narrow door, higher-loss walls everywhere else. Random motion on the flat part almost never stumbles through the door. Does the network ever reach the deeper solution, or is it stranded?
If the flat region is a symmetry, there is no door
First, a false alarm to rule out. If the flat region is a true gauge orbit, a symmetry, then it has no special door at all. A symmetry maps the whole neighbourhood of one point onto the neighbourhood of every other point on the orbit, carrying the loss values along exactly. So if there is a downhill ramp off the orbit at one spot, a copy of that ramp, perhaps stretched in raw coordinates but never walled off, exists at every spot. A genuine symmetry plateau cannot have a narrow one-percent entrance with cliffs elsewhere, because that would break the symmetry that defines it. From a gauge orbit you step off it from wherever you happen to sit, with no door to locate. This is also why a gauge-aware optimiser can safely treat such an orbit as flat and descend off it from any point.
So the hard case is a flat region that is not a symmetry: a real plateau. That is exactly the grokking picture.
The plateau and the door
On a memorising plateau the network already fits the training data, so the training loss is near zero and almost flat across a wide region of weights. Many different settings sit on it, all fitting the data, most generalising poorly. Among them is the setting that generalises, and it is usually the one with the smallest weights that still fit, a low-norm corner of the plateau. The puzzle is what carries the network there when the loss itself is flat.
The driver is the regulariser. Where the loss gradient is essentially zero, weight decay becomes the dominant signal, and it walks the weights steadily toward the smallest ones that still fit the data. The generalising solution is typically exactly that low-norm corner, so weight decay is what finds the door. How long the plateau lasts is how long weight decay takes to walk you there: flat for a long time, then a sudden drop. That is the standard account of why grokking is slow and then sudden, and it agrees with the weight-decay accounting of the previous explainer#18: hold the weight decay equal and a plain optimizer groks at very nearly the same step, because the decay is the engine.
Two things sharpen it. The loss is rarely perfectly flat, so even a faint tilt toward the exit, integrated over thousands of steps with momentum, adds to the drift. And adaptive optimisers amplify the effect: dividing each step by a tiny second moment turns a faint gradient on a flat region into a large step, so they cross plateaus faster than plain gradient descent.
Gauge-aware optimisers waste no motion
A flat region is usually part symmetry and part real plateau. A vanilla optimiser spends some of its motion wandering the symmetric part, which goes nowhere, since every point there is the same function. An optimiser aware of the gauge spends nothing on the symmetric directions and all of its motion on the real drift toward the exit, cleaner progress to the same solution. The symmetric part never needed a door; only the real plateau does.
The seeds tell it
If the decay is the engine and the gauge projection only steers, you would expect the projection to matter most where the walk is longest and most precarious, and that is exactly what the experiments show. On a shallow, well-tuned grid, a weight-decay-matched plain optimiser groks alongside the gauge-aware one, cell for cell. Deepen the network to 24 blocks, hold the weight decay equal, and the two come apart: the gauge-aware run clears the threshold on ten of eleven random seeds, while the plain one clears it on none, stalling at partial generalisation around 70% accuracy.2 Here are all twenty-two runs:
Depth multiplies the symmetric directions there are to wander, so the walk to the door gets harder to survive, and reliability, not raw speed, is what respecting the gauge buys.
So a solution is not stranded just because most of the landscape is flat. The objective and the regulariser together turn a flat plateau into a very gently tilted one and roll the network down to the low-norm exit, slowly, while the genuinely symmetric directions never needed an exit at all. The pathological case, perfectly flat with no regulariser, where you really would need luck, is real in principle. In trained networks the weight-decay pressure almost always supplies the signal that the bare loss surface hides.
Still open / underspecified. The account is the standard one, and its edges are live research. Why the one seed in eleven still fails at depth 24, how wide the regime is where the projection's reliability edge holds before the task gets easy enough that everything groks, and how much of the plateau's faint tilt is loss versus decay on real tasks, are open. The flat-landscape picture explains the shape of the curve; the quantitative theory of its timing is not settled.
- Grokking was named and first characterised in Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra, Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets, arXiv:2201.02177 (2022). ↩︎
- The depth-24 seed-reliability comparison at matched weight decay, and the weight-decay attribution of grokking speed, are from Tejas Pradeep Shirodkar, Dead-Direction Conditioners: Gauge-Equivariant Preconditioning for Deep Networks, arXiv:2606.29176 (2026). ↩︎