On the Hypersphere: μP Scaling of Optimizers with the Hyperball Mechanism

In modern neural network training, transferring hyperparameters across model scales has always been a core problem. For architectures with normalization (e.g., RMSNorm), the key is no longer the parameters themselves, but the evolution of features on the hypersphere [2].

Since normalized features satisfy $\lVert z \rVert_2 = \sqrt{n}$, when aligning across widths there is no longer any need to consider the feature norm itself; the components of $z$ already satisfy $\lvert z_i \rvert = \Theta(1)$. We only need to ensure:

Ensure that the evolution rate of the normalized feature $z$, $\lvert \left(\frac{dz}{dt}\right)_i \rvert = \Theta(1)$, remains at a stable magnitude.

Below, we first explain the problems with standard optimizers, then derive the scaling laws for the Hyperball family of optimizers proposed by Wen et al. [1]. If you haven’t read the previous article in this geometric line, you can first read “On the Hypersphere: From Spherical Dynamics to μP”; that article establishes the basic correspondence between spherical dynamics and $\mu$P learning rate scaling in RMSNorm architectures, without the Hyperball constraint.

1. Basic Setup and Continuous-Time Spherical Mapping

Let the hidden width of the network be $n$. The coordinate and norm magnitudes of key variables are as follows.

The input satisfies $\lVert x \rVert_2^2 = \Theta(n)$, with coordinate components of magnitude $x_j = \Theta(1)$.

Define the unnormalized feature $y_t = W_t x$. Under the assumption that the input directions are isotropically dispersed in the subspace of $W_t$, the squared $L_2$ norm of $y_t$ is:

From $\lVert x \rVert_2^2 = \Theta(n)$ and $\text{Tr}(W_t^T W_t) = \lVert W_t \rVert_F^2$, we obtain:

Thus:

After applying RMSNorm, the feature passed downstream is:

The corresponding Jacobian matrix is:

where $P_y$ is the orthogonal projection onto the hyperplane normal to $y_t$. Introducing the update matrix $U_t$, the spherical dynamics of the feature can be written as:

Requiring the coordinate components to align to $\Theta(1)$ yields the general alignment equation without norm constraints:

2. Intrinsic Radius Dependence and the Dynamical Equilibrium Dilemma of Standard Optimizers [2]

From the equation above, it is clear that what truly determines the spherical angular velocity of the feature is not the base learning rate $\eta$, but the effective spherical step size $\eta_{\mathrm{eff}}^{(i)}(t)$:

Here, the Frobenius norm of the weight matrix $\lVert W_t \rVert_F$ acts as the intrinsic radius controlling the spherical angular velocity. For standard optimizers with decoupled weight decay, this radius evolves according to a difference equation. Consider the stochastic gradient descent update with weight decay coefficient $\lambda$:

Taking the squared Frobenius norm on both sides of the above and expanding:

The normalization mechanism endows the weight matrix with scale invariance [2]; the network output does not change with the weight magnitude, so the gradient tensor is orthogonal to the current weight tensor, i.e., $\langle W_t, \frac{\partial \mathcal{L}}{\partial W_t} \rangle = 0$. Thus the cross term is zero:

To eliminate the dependence of the gradient norm on the current weight norm, introduce the unit gradient $\tilde{G}_t = \lVert W_t \rVert_F \frac{\partial \mathcal{L}}{\partial W_t}$. Substituting and taking the square root on both sides, and performing a Taylor expansion under $\eta \lambda \ll 1$, we obtain the master equation for the evolution of the intrinsic radius in a single step:

At the stationary state, the expectation of the intrinsic radius remains unchanged, i.e., $\mathbb{E}[\lVert W_{t+1} \rVert_F] = \mathbb{E}[\lVert W_t \rVert_F]$. Let $L = \mathbb{E}[\lVert \tilde{G}_t \rVert_F^2 \mid W_t]$ be the expected squared unit gradient norm. Setting the increment to zero yields the asymptotic limit $w^*$:

To analyze the dependence of $w^*$ on width $n$, we first need the magnitude of the norm of the unit gradient $\tilde{G}_t$. Let the gradient of the loss with respect to the normalized feature $z$ be $g_z$. By the chain rule, the gradient received by the unnormalized feature $y_t$ is $\frac{\partial \mathcal{L}}{\partial y_t} = \frac{\sqrt{n}}{\lVert y_t \rVert_2} P_y g_z$. Then, taking the gradient with respect to the weight tensor $W_t$ and substituting into the definition of $\tilde{G}_t$, combined with the identity $\lVert y_t \rVert_2 = \frac{\lVert x \rVert_2}{\sqrt{n}} \lVert W_t \rVert_F$ under the isotropy assumption, we have:

Its squared Frobenius norm is:

Assume the upstream normalized gradient satisfies $(g_z)_i = \Theta(1)$, then $\lVert g_z \rVert_2^2 = \Theta(n)$. Since $P_y$ is an orthogonal projection onto a hyperplane, it only attenuates the norm by a factor of $\Theta(1)$, so $\lVert P_y g_z \rVert_2^2 = \Theta(n)$. Thus, the expected squared unit gradient norm $L$ scales as:

Substituting the magnitude of $L$ into $w^*$, and setting the weight decay coefficient $\lambda = \Theta(1)$, we obtain the intrinsic radius scaling at the stationary point:

To satisfy the alignment criterion $\lvert \left(\frac{dz}{dt}\right)_i \rvert = \Theta(1)$, consider the stochastic gradient descent update direction $U_t = \frac{\partial \mathcal{L}}{\partial W_t} = \frac{\sqrt{n}}{\lVert y_t \rVert_2} (P_y g_z) x^T$. Its action on the input feature $x$ is:

Substituting $\lVert x \rVert_2^2 = \Theta(n)$, $\lVert y_t \rVert_2 = \Theta(\lVert W_t \rVert_F)$, and using $P_y^2 = P_y$:

Thus, the magnitude of the absolute value of the coordinate component is $\lvert (P_y (U_t x))_i \rvert = \Theta\left(\frac{n \sqrt{n}}{\lVert W_t \rVert_F}\right)$. Substituting into the feature evolution alignment equation:

Substituting the intrinsic radius scaling at the stationary point $\lVert W_t \rVert_F = \Theta(n^{3/4} \eta^{1/4})$ into the above constraint:

Solving yields the learning rate scaling law required for cross-scale alignment:

Substituting back into the intrinsic radius expression, we obtain the norm magnitude of the system in the dynamical equilibrium state:

This shows that if the system can instantaneously reach dynamical equilibrium, then when $\eta = \Theta(1/n)$, the intrinsic weight norm of standard optimization will automatically converge to $\Theta(\sqrt{n})$, consistent with the magnitude corresponding to standard initialization variance. Its core insight is the same as that of μP.

However, in practical engineering, relying on this mechanism to approach the natural stationary point in order to maintain hyperparameter alignment runs into three problems:

1. Convergence delay: $w^*$ is an asymptotic limit, and the weight norm does not instantly reach the equilibrium point. In the early training phase, $\lVert W_t \rVert_F$ has not yet converged to $\Theta(\sqrt{n})$, so the effective step size on the hypersphere is incorrect.
2. Imbalance upon learning rate changes: Modern training commonly uses multi-stage learning rate schedules. Once $\eta$ decays, the corresponding equilibrium point $w^*$ immediately changes, but the weight norm takes time to catch up with the new equilibrium point. During this transition period, the alignment condition does not hold.
3. Orthogonality assumption does not always hold: The above derivation relies on $\langle W_t, \frac{\partial \mathcal{L}}{\partial W_t} \rangle = 0$, i.e., the gradient is strictly orthogonal to the weights. However, in networks with residual connections, this condition is not strictly satisfied; the cross term is nonzero, and the weight norms of each layer drift independently, preventing unified alignment.

3. Hyperball constraint and unified master equation

To fundamentally eliminate the interference of the intrinsic radius $\lVert W_t \rVert_F$ on the dynamics, the Hyperball mechanism proposed by Wen et al. [1] explicitly introduces the tangent space projection operator $\Pi_W$, constraining the weight update direction to the hypersphere with the initial norm as radius.

In the continuous-time limit of the discrete update rule, the weight dynamics equation is:

Under standard initialization, the matrix element sampling variance is $\frac{1}{n}$, so the initial constant $R = \lVert W_0 \rVert_F = \Theta(\sqrt{n})$. The tangent space projection operator $\Pi_W$ ensures that the Frobenius norm of the weight matrix remains constant at any time $t$:

Therefore, the dynamic denominator becomes an invariant in time and width:

Using the linearity property of the projection operator $P_y(\Pi_W(A)x) = P_y(Ax)$, and substituting back into the Jacobian matrix from Section 1, the prefactor simplifies to the constant $\frac{\sqrt{n}}{\Theta(\sqrt{n})} = \Theta(1)$. Thus, we obtain the unified master equation governing the dynamics of all Hyperball variants:

For $\lvert \left(\frac{dz}{dt}\right)_i \rvert = \Theta(1)$ to hold, the learning rate $\eta$ must satisfy:

This transformation decouples the alignment of hyperparameters from the convergence state of the system. Without Hyperball, to obtain the correct spherical angular velocity, the network must rely on weight decay and gradient orthogonality to reach the equilibrium state $\lVert W_t \rVert_F = \Theta(\sqrt{n})$; with Hyperball, $\lVert u_t \rVert_F$ is explicitly placed in the numerator of the learning rate, so the setting of $\eta$ depends only on the gradient structure of the current update step, and is no longer constrained by the historical trajectory of the weight norm.

4. Alignment derivations for specific Hyperball variants

Below we analyze the update characteristics of different optimizers under specific assumptions and give their respective alignment learning rates.

4.1 Alignment derivation for SGDH

Assume the upstream gradient $g = \nabla_z L$ has coordinate components of magnitude $\Theta(1)$. The base gradient update matrix is $u_t = \Theta(1) (P_y g) x^T$. Since $\lVert P_y g \rVert_2 = \Theta(\sqrt{n})$ and $\lVert x \rVert_2 = \Theta(\sqrt{n})$, we have:

Now consider the action of the update matrix on the input feature vector:

After applying the orthogonal projection operator:

From the assumption above that $\lvert (P_y g)_i \rvert = \Theta(1)$, we have $\lvert (P_y (u_t x))_i \rvert = \Theta(n)$. Substituting into the master equation:

4.2 Alignment derivation for AdamH

For the assumption $\lVert u_t \rVert_F = \Theta(n)$ used by both AdamH and MuonH below, we keep only the result itself. A more detailed Frobenius norm estimate can be found in “Frobenius Norm Estimates for the Update Matrices of Adam and Muon Optimizers”, which devotes a separate section to this step.

After extracting the sign matrix of the gradient, the update matrix $u_t$ contains $n^2$ elements with absolute value $1$, so its norm is:

When momentum or large batches are introduced, the signs of the update matrix and the coordinate distribution of the input vector $x$ are approximately independent. By the central limit theorem, the linear combination of $n$ independent terms gives $\lvert (u_t x)_i \rvert = \Theta(\sqrt{n})$, so $\lvert (P_y (u_t x))_i \rvert = \Theta(\sqrt{n})$. Substituting into the master equation:

4.3 Alignment derivation for MuonH and its isotropy advantage

In the actual engineering implementation of Muon, the orthogonalized update is further adjusted by a learning rate so that the root mean square magnitude of the update across different matrix shapes matches that of standard optimizers.

Under this setting, the leading order of the Frobenius norm of the update matrix is:

Now consider the action of this update matrix on the current input feature $x$. Due to orthogonalization and root mean square alignment, after tangent space projection, the typical magnitude of each coordinate component is:

Substituting into the master equation yields the optimal learning rate:

This shows that the root cause of drift in unconstrained Muon is that the intrinsic radius cannot automatically align across different widths; after MuonH enforces $\lVert W_t \rVert_F = \Theta(\sqrt{n})$, the leading term aligns to a constant learning rate.

Further comparing MuonH and AdamH, MuonH often provides more precise cross-scale alignment. The reason is that even with the Hyperball constraint, Adam’s update matrix $u_t$ still relies on elementwise adaptive normalization, so the projected component $\lvert (P_y (u_t x))_i \rvert$ retains residual anisotropy; whereas Muon’s orthogonalization flattens the singular value structure of the update matrix, making its angular action on the current feature more isotropic. Therefore, after the norm is explicitly fixed, MuonH has smaller residual alignment error across different widths.

5. Summary of Global Scaling Laws

Based on the above master equation in feature space, the learning rates $\eta$ required by the Hyperball family of optimizers to achieve feature angular velocity alignment under different geometric and statistical assumptions are summarized as follows:

6. Conclusion

Traditional optimization relies on the intrinsic weight norm to find natural equilibrium points, but this mechanism is deeply coupled with network width, scheduling strategy, and model architecture, so it cannot guarantee cross-scale consistency of spherical angular velocity when model scale changes. Hyperball eliminates this intrinsic dependence through geometric projection constraints on the hypersphere, simplifying the Jacobian prefactor to a scalar constant. The derivation shows that only by taking $\lvert \left(\frac{dz}{dt}\right)_i \rvert = \Theta(1)$ as the unified alignment criterion and cutting off the coupling between the intrinsic weight norm and hyperparameters can the scaling law of optimizer hyperparameters be clearly characterized. If you wish to fill in the update matrix norm assumptions used for AdamH / MuonH in this article, you can go on to read “Frobenius Norm Estimates of the Update Matrices of Adam and Muon Optimizers”.

References

[1] Wen, K., Dang, X., Lyu, K., Ma, T., & Liang, P. (2025). Fantastic Pretraining Optimizers and Where to Find Them 2.1: Hyperball Optimization. https://tinyurl.com/muonh

[2] Wan, R., Zhu, Z., Zhang, X., & Sun, J. (2020). Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGD. arXiv preprint arXiv:2006.08419. https://arxiv.org/abs/2006.08419

Citation

If you need to cite this article, please refer to:

@article{zou2026sphericalhyperball,
  title={球面之上：带有 Hyperball 机制的优化器的 μP 缩放},
  author={Zou, Jiaxuan},
  journal={Jiaxuan's Blog},
  year={2026},
  url={[https://jiaxuanzou0714.github.io/blog/2026/spherical-hyperball/](https://jiaxuanzou0714.github.io/blog/2026/spherical-hyperball/)}
}