Bias and Fluctuations of the Spectral Norm of Random Gaussian Matrices at Finite Width

Preface: In the previous blog posts on muP, we often had to deal with quantities such as random matrices, spectral norms, or Frobenius norms. For these quantities, a core insight of Tensor Programs is that in the large-width limit, the asymptotic behavior of these quantities is often very stable. The author, Greg Yang, said that when characterizing scaling laws, we actually want to characterize the behavior of the network in the limit state (width/depth/training time limits), so we naturally use the law of large numbers and the central limit theorem to analyze the limiting behavior of these quantities. However, I believe that finite-width networks inevitably introduce some systematic biases and random fluctuations, which are ignored in the large-width limit but may be very important in actual networks (for example, they may affect our muP scaling or optimizer design). This article takes the spectral norm as an example to analyze its behavior at finite width.

In many width-scaling theories, we are accustomed to treating the spectral norm of a randomly initialized matrix as a stable $\Theta(1)$ quantity; for example, when the matrix elements follow a Gaussian distribution with zero mean and variance $1/n$, intuitively its spectral norm should be “close to a constant.” But if we truly care about finite-width networks, we cannot stop at the large-width limit, because for random matrices, the spectral norm itself is not a deterministic quantity without fluctuations. Not only is it random, but that randomness does not follow the most common central limit theorem scaling; instead it has a finer edge-fluctuation structure.

This article starts from the matrix form and discusses an $n\times n$ Gaussian random matrix

that is, the case $W_{ij}\sim\mathcal N(0,1/n)$. Our goal is to estimate the behavior of its spectral norm $|W|_2$ at finite width and decompose it into three parts: the macroscopic leading limit, the finite-width bias, and the finite-width random fluctuation.

1. From the spectral norm to the Wishart matrix

The most natural entry point for the spectral norm is the covariance matrix:

Thus

This step is crucial. Once we switch to $S$, the problem becomes the classic largest-eigenvalue problem for a Wishart matrix, which is precisely one of the most thoroughly developed objects in random matrix theory.

In the large-width limit $n\to\infty$, the empirical spectral distribution of $S$ follows the Marchenko–Pastur law. For the square case here, the aspect ratio is $1$, and the spectral support is $[0,4]$. Furthermore, we have

Therefore

This gives the most common zeroth-order conclusion: when the width is large enough, for a Gaussian matrix with element variance $1/n$, the leading order of its spectral norm is stable around $2$. Many scaling analyses are built on exactly this $\Theta(1)$ macroscopic scale.

2. Beyond the macroscopic limit: edge fluctuations of the largest eigenvalue

But “tends to 2” does not mean “equals 2.” If we care about finite-width corrections, we must study how $\lambda_{\max}(S)$ fluctuates near the edge $4$.

For a standard real Wishart matrix $X^\top X$, Johnstone (2001) proved: if

then

In the square case (p=n) we care about here,

Therefore, equivalently, we can write

where $TW_1$ denotes the order-1 Tracy–Widom distribution.

Since $S=\frac1n X^\top X$, rewriting the above formula for $S$ gives

Thus

\[\lambda_{\max}(S)=4+2^{4/3}n^{-2/3}\xi_n+o_p(n^{-2/3}),\qquad \xi_n \Rightarrow TW_1.\]

This formula already reveals the most important point in finite-width analysis: the correction scale of the largest eigenvalue is not the common $n^{-1/2}$ from the CLT, but the finer $n^{-2/3}$.

3. Finite-width expansion of the spectral norm

What we really care about is the spectral norm itself, not $\lambda_{\max}(S)$. Therefore we need to propagate the above result back to

Perform a Taylor expansion of the function $f(x)=\sqrt x$ around $x=4$:

Let

Substituting gives

That is,

This is the finite-width expansion we wanted. It is richer than the simple statement “the spectral norm is approximately 2” because it tells us:

1. The limit is $2$;
2. The first-order correction is of order $n^{-2/3}$;
3. This correction term is itself random and follows Tracy–Widom-type statistics.

Furthermore, the mean and variance of the Tracy–Widom distribution for $\beta=1$ are

Then we immediately obtain

\[\|W\|_2 = \underbrace{2}_{\text{macroscopic limit}} + \underbrace{2^{-2/3}\mu_{TW}n^{-2/3}}_{\text{finite-width bias}} + \underbrace{2^{-2/3}n^{-2/3}(\xi_n-\mu_{TW})}_{\text{finite-width fluctuation}} + o_p(n^{-2/3}).\]

Taking expectations gives

and

These two conclusions correspond respectively to the systematic bias and random fluctuation at finite width, and are exactly what the numerical verification below will check.

4. Numerical verification

The theoretical conclusions above can be checked directly with numerical experiments. For each width $n$, independently generate Gaussian random matrices multiple times

and record the spectral norm of the $k$-th experiment as

The three statistics in the figure are all computed from this sample set ${s_k}_{k=1}^M$:

Here bias is defined as the “positive deviation relative to the infinite-width limit $2$”, i.e., how much the mean is below $2$; therefore the bias in the figure is positive, and theoretically satisfies

while the standard deviation satisfies

The left, middle, and right panels of the figure below correspond to $\mathrm{Mean}(n)$, $\mathrm{Bias}(n)$, and $\mathrm{Std}(n)$, respectively. Thus this set of figures does not merely show loosely that “the spectral norm converges”, but separately tests three things: the mean approaches $2$ from below, the systematic bias decays as $n^{-2/3}$, and the standard deviation of the random fluctuations decays at the same scale.

These plots also make it easier to see that the finite-width effect is not simply a matter of “a bit more random noise”. The mean curve itself shows a stable downward bias.

5. A summary formulation better suited to subsequent modeling

If this result is to be plugged into the network forward pass, normalization, or optimization dynamics, a convenient expression is:

where

The advantage of writing it this way is that later, if some quantity depends on the random matrix $|W|_2$, it can be directly decomposed into:

1. the macroscopic limit term;
2. the deterministic shift due to finite width;
3. the random perturbation due to finite width.

References

[1] V. A. Marchenko and L. A. Pastur, Distribution of eigenvalues for some sets of random matrices, Mathematics of the USSR-Sbornik, 1967.

[2] Z. D. Bai and Y. Q. Yin, Limit of the smallest eigenvalue of a large dimensional sample covariance matrix, Annals of Probability, 1993.

[3] I. M. Johnstone, On the distribution of the largest eigenvalue in principal components analysis, Annals of Statistics, 2001.

[4] C. A. Tracy and H. Widom, Level-spacing distributions and the Airy kernel, Communications in Mathematical Physics, 1994.

Citation

If you need to cite this article, please refer to:

@article{zou2026finite-width-spectral-norm,
  title={有限宽度下随机高斯矩阵谱范数的偏置与涨落},
  author={Zou, Jiaxuan},
  journal={Jiaxuan's Blog},
  year={2026},
  url={https://jiaxuanzou0714.github.io/blog/2026/finite-width-spectral-norm/}
}