These are English translations of posts originally written in Chinese. Each one links back to its source. The originals live on the main blog page.
-
Hyperball, effective lr, and the shape of peak-then-decay
Two questions. What is scheduled: changes in the weight norm superimpose an implicit schedule on top of the base lr schedule; Hyperball removes this layer, and the quantity that matters is the effective lr. Shape: peak-then-decay is the solution to the bias–variance trade-off, with the bias term dominating early and the variance term dominating late; the shapes that satisfy this balance form a set and are not limited to a specific analytic form.
-
My deepest insight after two years of undergraduate study: cognitive compound interest
Many people are used to making long-term linear plans with their current low-order cognition, falling into anxiety about not seeing returns from short-term efforts. This article, based on my real growth experience over two years of undergraduate study, deconstructs the underlying logic and practical path of "cognitive compound interest," explaining that only continuous output, heavy investment in high-density environments, and high-frequency iteration of cognition can achieve the snowball effect of growth.
-
DASF: A Closed-Loop Schedule-Free Method for Batch Size
This post proposes DASF (Drift-Aware Schedule-Free): based on the duality of Schedule-Free (iterate averaging ↔ learning rate schedule, gradient averaging ↔ batch size schedule), it uses on-the-fly measured gradient statistics to set the effective batch size online, with no schedule and no tuning, eliminating the cost of training proxy models and fitting scaling laws for batch calibration. On real transformers it matches or exceeds tuned baselines, and provides a falsifiable negative result: under compute-optimal conditions, the optimal effective batch is approximately constant, not growing as √t.
-
Why does the batch size need to be doubled midway through LLM pretraining?
Starting from the Double GBS phenomenon of Apertus 70B, we use the gradient noise scale, critical batch size, and calculus of variations to derive the optimal schedule for increasing the batch size midway through LLM pretraining, and validate it on the noisy quadratic model.
-
Don't Just Learn from the 19th-Century West: Civilizational Centrism, Cosmopolitanism, and the Public Responsibility of Young Leaders
After reading Liu Qing's article on tianxia and new cosmopolitanism, here are some of my thoughts on civilizational centrism, 19th-century-style imaginings of great power, and the public responsibility of young technologists.
-
Re-listening to Yang Zhilin: Bet on Scaling, First Principles, and Long-termism
At lunch, I re-listened to the conversation between Yang Zhilin and Zhang Xiaojun from January 2024, jotting down some thoughts on long context, scaling laws, agents, AGI organizational forms, and long-termism.
-
μP Map
A reading guide and thread overview for the μP-related blog posts.
-
In the LLM context, how does noise in the gradient affect training dynamics?
A discussion of gradient noise in the late stages of LLM pretraining, and why blockwise normalized updates look more like constraining the update magnitude than correcting the gradient direction.
-
The Trade-off Between Parallelism and Expressiveness: Theoretical Boundaries from $AC^0$/$TC^0$ to Linear Attention
From the perspective of circuit complexity, this post gives a unified explanation of why constant-depth Transformers cannot exactly perform integer multiplication of arbitrary length, and why stronger linear attention variants often fail to maintain full token parallelism.
-
Bias and Fluctuations of the Spectral Norm of Random Gaussian Matrices at Finite Width
Starting from Wishart random matrix theory, this article derives the expansion of the spectral norm of a Gaussian matrix with element variance 1/n at finite width, showing that it not only converges to the macroscopic limit 2, but also carries a bias of order $n^{-2/3}$ and Tracy-Widom type random fluctuations.
-
Frobenius Norm Estimation of the Update Matrices for Adam and Muon Optimizers
This article rigorously derives and estimates the Frobenius norm of the update matrices of the Adam and Muon optimizers in a single iteration step, and explores the influence of matrix shape on the order of magnitude of the norm.
-
On the Hypersphere: μP Scaling of Optimizers with the Hyperball Mechanism
Starting from the first principles of continuous-time spherical dynamics, this article explores how the intrinsic dependence of weight norms disrupts hyperparameter alignment, and rigorously derives the underlying mathematical mechanisms by which various Hyperball optimizer variants achieve feature-space alignment.
-
On the Hypersphere: From Spherical Dynamics to μP
This article departs from the probabilistic framework of Tensor Programs and, from the perspective of continuous-time spherical dynamics, rigorously derives how to achieve alignment of networks of different sizes by aligning the dynamics on the hypersphere in network architectures that apply RMSNorm.
-
On the Loose Use and Methodological Boundaries of the "Manifold" Concept in Current AI
This article discusses the loose use of the "manifold" concept in AI theory research and draws the boundaries between engineering terminology, geometric intuition, and rigorous mathematical argument.
-
Tensor Programs (Part 2): From Tensor Programs to μP
This article systematically reviews the core theoretical derivations of the maximal update parameterization (μP) obtained from Tensor Programs. The most fundamental and central insight of Tensor Programs theory in deriving neural network scaling laws is that one must strictly distinguish and apply the law of large numbers (LLN) and the central limit theorem (CLT) according to the different generation mechanisms of weight tensors.
-
Tensor Programs (Part 1): From the Spectral Condition of Feature Learning to μP
This article introduces the introductory paper of Greg Yang's Tensor Programs series—A Spectral Condition for Feature Learning—which derives the scaling conditions required for feature learning from the perspective of the spectral norm, and then re-derives the Maximal Update Parametrization (μP) from them.
-
From Gated DeltaNet to Kaczmarz
This article starts from the online learning formulation of Gated DeltaNet, introduces the Kaczmarz algorithm as an alternative to SGD, and analyzes its geometric meaning and its connection to Longhorn.
-
How to align data scaling curves under different initialization scales
We study the relationship between the empirical slope of data scaling and the initialization std, and propose a simple method to align data scaling curves under different initialization scales.