μP Map

This post is devoted to laying out the thread that runs through the μP-related articles. Rather than viewing them as several scattered blog posts, I prefer to see them as several complementary derivation routes around the same core question: why large models can still maintain feature learning as width is scaled up, and how the corresponding initialization, learning rate, and optimizer scaling rules should be understood.

Core Question

These articles largely revolve around the following question:

1. When model width increases, how to maintain feature learning rather than degenerate into lazy learning.
2. If we want hyperparameters tuned on small models to transfer to large models, how should the scaling rules be derived.
3. For different architectures and different optimizers, which quantities are the ones that should actually be aligned.

Overall Thread

If we compress the entire set of articles into a single main line, its structure is roughly as follows:

1. Background problem: first explain why scaling laws, feature learning, and width scaling are worth studying.
2. Tensor Programs route: derive μP from spectral conditions, LLN/CLT, and parameterization rules.
3. Spherical dynamics route: under RMSNorm architectures, reinterpret the alignment problem as dynamical alignment on a hypersphere.
4. Hyperball extension: further incorporate optimizers and norm constraints into the same geometric framework.
5. Technical supplement: separately fill in the update matrix norm estimates used but not elaborated in the Hyperball article.

Recommended Reading Order

1. Background and Phenomena

This part is now better read as an entry point through the paper: first look at Effective Frontiers: A Unification of Neural Scaling Laws.

This paper essentially inherits the problem framing of the original blog post Can We Derive Scaling Law From First Principles?, namely the big question of “why scaling laws can be explained from first principles.” In this reading path, it serves better as the formal background entry, while the old post can be seen as an earlier landing point on the web.

Next, you can read “How to Align Data Scaling Curves under Different Initialization Magnitudes”.

This one leans toward empirical phenomena, discussing why data scaling curves shift when the initialization standard deviation changes. Its relationship with the later μP articles is: the former shows the phenomenon, while the latter attempts to explain what kind of scaling keeps the learning dynamics consistent across different widths.

2. Tensor Programs Route

This line is suitable for readers who want to understand μP from Greg Yang’s original theoretical framework.

The first article is “Tensor Programs (Part 1): From the Spectral Conditions of Feature Learning to μP”.

The focus of this article is not to directly give a set of parameterization rules, but to first answer a more basic question: why do we need μP. It starts from the conditions for maintaining feature learning, and uses the perspective of spectral norm and one-step updates to explain which quantities must remain stable when width is scaled.

The second article is “Tensor Programs (Part 2): From Tensor Programs to μP”.

This one can be seen as taking the first one further. It unpacks the key statistical insight of the Tensor Programs route: initialization is dominated by CLT, while gradient updates are dominated by LLN. It is also here that the initialization scaling and learning rate scaling of μP are more systematically organized.

If you prefer to finish the Tensor Programs line first before moving to the geometric route, it is most natural to read these two and then proceed.

3. Spherical Dynamics Route

If you care more about “what is actually being aligned” under the RMSNorm architecture, you can start with “On the Hypersphere: From Spherical Dynamics to μP”.

The importance of this article is that it does not continue the probabilistic limit theorem language of Tensor Programs, but instead takes a different perspective: under RMSNorm, what really needs to be aligned is not the parameter norm itself, but the evolution rate of features on the hypersphere. In other words, this article provides a derivation path for μP that runs in parallel with Tensor Programs but is more geometric.

In terms of reading relationships, this article can be placed after Tensor Programs (Part 2) as a comparative reading, or it can serve as a standalone starting point for the geometric route.

4. Hyperball and Optimizer Extensions

Building on the spherical dynamics route, the next natural article is “On the Hypersphere: μP Scaling of Optimizers with the Hyperball Mechanism”.

This article pushes the problem one step further: if we not only discuss SGD but also include optimizers like Adam and Muon together with norm constraint mechanisms such as Hyperball, how do the cross-scale alignment conditions change? Its core role is to advance “spherical dynamics” from basic geometric analysis to the level of “how different optimizers scale.”

If “On the Hypersphere: From Spherical Dynamics to μP” mainly addresses the foundational framework, then this article addresses the generalization problem at the optimizer level.

5. Supplementary Notes on Norm Estimation

Finally, read “Frobenius Norm Estimation of the Update Matrices for Adam and Muon Optimizers”.

This article is more like a “technical note.” In the Hyperball article, the derivations for AdamH and MuonH both use the order-of-magnitude conclusion $\lVert u_t \rVert_F = \Theta(n)$; the role of this article is to isolate this conclusion and specifically discuss why the Frobenius norm of the update matrices for Adam and Muon falls into this order of magnitude.

Therefore, it is best read after the Hyperball article. When read this way, you will clearly see that it is not a standalone post written from scratch, but rather fills in a technical premise that was temporarily compressed away in the Hyperball derivation.

The Relationship Between the Two Main Lines

If we keep only the essential structure, the μP-related posts can currently be divided into two main lines.

Main Line A: Tensor Programs Perspective

1. “Tensor Programs (Part 1): From the Spectral Conditions of Feature Learning to μP”
2. “Tensor Programs (Part 2): From Tensor Programs to μP”

This line emphasizes: understanding why μP holds through width limits, spectral conditions, LLN/CLT, and the formal tools of Tensor Programs.

Main Line B: Spherical Dynamics Perspective

1. “On the Hypersphere: From Spherical Dynamics to μP”
2. “On the Hypersphere: μP Scaling of Optimizers with the Hyperball Mechanism”
3. “Frobenius Norm Estimation of the Update Matrices for Adam and Muon Optimizers”

This line emphasizes: under the settings of RMSNorm and norm constraints, reinterpreting the scaling problem as a problem of geometric dynamics and optimizer update structure.

How to Use This Reading Guide

If this is your first time reading these articles, I recommend simply following the “Recommended Reading Order” from start to finish.

If you already have a rough idea of what μP is but want to fill in the theoretical details, you can jump directly to any of the following branches:

1. To follow Greg Yang’s original theoretical route: start with “Tensor Programs (Part 1): From the Spectral Conditions of Feature Learning to μP”.
2. To follow the RMSNorm and geometric dynamics route: start with “On the Hypersphere: From Spherical Dynamics to μP”.
3. To go straight to the optimizer-level generalization: start with “On the Hypersphere: μP Scaling of Optimizers with the Hyperball Mechanism”.

If more μP-related blog posts are added in the future, this guide will be updated accordingly.