Hyperball, effective lr, and the shape of peak-then-decay

Created on August 25, 2026   ·   -- views

2026   ·   deep-learning   lr-schedule   optimizer   spherical-dynamics   scaling-law   ·   deep-learning

阅读中文原文

TL;DR

Two phenomena have appeared in several recent works. First, Hyperball fixes the Frobenius norm of the weight matrix and its update to a constant, achieving a 20–30% token-equivalent speedup over the weight decay baseline [8]. Second, methods whose settings differ from one another all yield optimal learning rate curves that are peak-then-decay.

Peak-then-decay in this article refers to the following shape: the learning rate rises to a peak $\eta_{\max}$ in the early phase of training, then monotonically decreases, ending close to zero. The functional form of the decay segment is unrestricted.

peak plus decay learning rate shape illustrated with three power-law exponents
Examples of peak-then-decay shapes, where the decay segment takes the power-law form $\eta_t\propto(1-t/T)^{\alpha}$, with $\alpha$ equal to 0.5, 1, and 2. $\alpha=0.5$ corresponds to the sqrt shape, and $\alpha=1$ corresponds to linear decay to zero. The three curves have the same peak and endpoint, but the concavity of the decay segment differs.

The power-law form in the figure is convenient for parameterization, and manually specified schedules often adopt this form. Among the four works listed in Section 4, the refined schedule is constructed pointwise from the gradient norm sequence, and the actual learning rate of Schedule-Free+ is produced by iterate averaging; both curves are peak-then-decay and do not correspond to a specific analytic form.

Several recent works attribute the source of the above two phenomena to the same quantity.

Core Judgment

The quantity that determines training progress is the effective learning rate \(\eta_t^\star=\eta_t\lVert U_t\rVert/\lVert W_t\rVert\).
The weight norm grows continuously during training, so the conversion factor from \(\eta_t\) to \(\eta_t^\star\) changes continuously, superimposing an implicit schedule on top of the specified learning rate schedule. Hyperball fixes the weight norm, removing this layer, and its effect is therefore an implicit learning rate schedule.

The reason $\eta^\star$ takes the peak-then-decay shape is independent of the optimizer:

Shape Criterion

Peak-then-decay is the solution to the bias–variance trade-off.
The peak segment reduces bias with larger step sizes, and the end decays to zero to reduce variance. Shapes satisfying this balance form a set, and the performance of shapes within the set is comparable.

The two judgments give the structure of this article: $\eta^\star$ is the quantity that should be scheduled (Sections 1–3), and the bias–variance trade-off determines its shape (Section 4).

1. The Quantity That Matters Is the Effective Learning Rate

Parameters followed by normalization layers satisfy scale invariance $\mathcal{L}(\rho W) = \mathcal{L}(W)$. Pure radial scaling does not change the network function, so the quantity characterizing single-step progress is the angle through which the weight direction turns. Hunyuan ELR calls this quantity the angular update size (AUS) [1]:

\[\mathrm{AUS} := \left\lVert \frac{W_{t+1}}{\lVert W_{t+1}\rVert} - \frac{W_t}{\lVert W_t\rVert}\right\rVert \approx \frac{\eta_t \lVert U_t\rVert}{\lVert W_t \rVert} =: \eta_t^\star\]

The right-hand side is the effective learning rate, which originates from an earlier line of dynamical analyses of weight decay [9]. It contains $\lVert W_t \rVert$, and the weight norm changes continuously during training, so the conversion factor from $\eta_t$ to $\eta_t^\star$ changes continuously. This conversion factor is the implicit schedule mentioned in the TL;DR.

The magnitude of the conversion can be given by a comparison of three nominal schedules: WSD (peak $3.6\times10^{-3}$), cosine and linear (peak $8.8\times10^{-3}$), with the peaks differing by a factor of about 2.4.

weight norm and angular update size under WSD, cosine and linear schedules
Bottom right: the learning rate schedules of WSD, cosine, and linear; top right: the corresponding weight norm; bottom left: the corresponding angular update size. Image source: Hunyuan ELR.

Comparing the bottom right and bottom left: the nominal schedule shapes differ significantly, but after 1000 steps the three angular update size curves nearly coincide, all being peak-then-decay. The top right gives the corresponding weight norm, which grows from about 30 to 180–280 in the first 2000 steps.

Nominal Learning Rate and Effective Learning Rate

The quantity being tuned is the nominal learning rate schedule, while the quantity that acts on training is the AUS. Between the two there is a conversion factor that varies with the weight norm.

Under a constant nominal learning rate, the form of this conversion is as follows [2].

nominal and effective learning rate of MuonWD and MuonH under a constant schedule
Left: nominal learning rates of MuonWD and MuonH, with the two curves coinciding at 0.025; right: the corresponding effective learning rates, where MuonWD decays from about $9\times10^{-5}$ to about $1.8\times10^{-5}$, and MuonH remains constant after the initial transient. Image source: Hyperball May Not Be a Free Lunch, Figure 2(a)(b).

The nominal learning rates are exactly the same, but the effective learning rates differ by a factor of about 5. The decay in MuonWD comes from weight norm growth. MuonH fixes the weight norm, and the effective learning rate remains constant after the initial transient.

2. Correspondence Between the Effective Learning Rate Trajectory and the Loss Curve

The previous section gave the difference between $\eta^\star$ and $\eta$. For $\eta^\star$ to serve as the object of analysis, one further point needs to be confirmed: whether the loss curves are the same when the $\eta^\star$ trajectories are the same. Two works have tested this from opposite directions.

AUS-replay in Hunyuan ELR: train GPT-2 (124M) with Adam or Muon and record the AUS step by step, then switch to the corresponding Hyperball variant (AdamH, MuonH) and set the recorded AUS trajectory as its learning rate curve.

AUS replay experiment comparing Adam/Muon with their Hyperball variants
First row: AUS curves; second row: validation loss; third row: loss difference. The four columns are results for Adam and Muon on naive nanoGPT and scale-invariant nanoGPT. Image source: Hunyuan ELR.

The AUS curves and loss curves of the two groups nearly coincide, and on scale-invariant structures the loss difference is within $\pm0.005$ [1].

The opposite approach is to fix the optimizer and adjust its learning rate step by step to match the target effective learning rate, aligning MuonWD to the trajectory of MuonH, and also aligning in the reverse direction [2].

mutual alignment of MuonWD and MuonH training loss by learning-rate alignment
Left: MuonH aligned to MuonWD, the green dashed line nearly coincides with MuonWD; right: MuonWD aligned to MuonH, the purple dashed line nearly coincides with MuonH. In both figures, the original curves of MuonWD and MuonH differ in the 500–2000 step range. Image source: Hyperball May Not Be a Free Lunch, Figure 3(a)(c).

The alignment results in both directions nearly coincide with the target curve, and changing the learning rate suffices to reproduce the loss curve of the other optimizer. The paper concludes that the main role of Hyperball is an implicit state-dependent learning rate schedule, and its update direction shows no additional advantage [2].

Verification results in both directions

When the \(\eta^\star\) trajectories are the same, the loss curves nearly coincide, and the difference between Hyperball and non-Hyperball optimizers can be explained by the learning rate alone.
This yields the opening judgment: Hyperball removes the implicit schedule that the weight norm superimposes on top of the learning rate schedule.

This judgment also matches the phase-dependent behavior of MuonH: convergence is slower early on, while accuracy later in training is higher than that of MuonWD [2]. Under the Hyperball constraint, there is no decay from weight norm growth, and the early actual step size is larger than that of MuonWD.

3. Fit accuracy of scaling laws with the effective learning rate

If $\eta^\star$ is the quantity that actually takes effect, using it to fit the loss curve should yield higher accuracy than using $\eta$. After Hunyuan ELR replaces $\eta$ with $\eta^\star$ in the multi-power law (MPL) loss model, both the in-sample fit and the cross-schedule prediction become more accurate, and the optimal $\eta^\star$ also transfers more reliably across model width and depth [1].

The reason is consistent with Section 1: between $\eta$ and the loss there is a conversion factor that varies with training and with width and depth, and $\eta^\star$ does not include this factor. When fitting the scaling law with $\eta$ as the independent variable, the variation of this factor is included in the fit error.

The same relationship applies to hyperparameter transfer. Under the Hyperball constraint, $\Delta\phi_t \approx \eta_t$, so

\[\sum_t \Delta\phi_t \approx \int_0^T \eta_t\,\mathrm{d}t\]

That is, the integral of the learning rate equals the total angle through which the weight direction turns. A related phenomenon is that when the cumulative learning rate of two training runs is similar, the final losses are similar [1]. Under the Hyperball constraint, this quantity has a corresponding geometric interpretation and can serve as the quantity to align when transferring across budgets.

Another related result: on the Frobenius sphere, weight decay has no effect at first order, reducing the two-dimensional search over $(\eta,\lambda)$ to a one-dimensional search. Measurements give the optimal learning rate as a power law in the number of tokens, $\eta^*\propto T^{-0.32}$, consistent with the exponent reported for AdamW [3]. The theoretical origin of this exponent has not been identified.

4. Source of the shape: bias–variance trade-off

The first three sections identify the quantity being scheduled; this section discusses the shape of that quantity. The settings of the following four works differ, but the resulting learning rate curve shapes are consistent: rising to a peak in the early phase of training, then monotonically decreasing to near zero.

WSD cooldown shape comparison. Comparing manually specified shapes within the cooldown segment, sqrt ($1-\sqrt{x}$) and lowered linear 0.7 perform comparably, with lowered linear 0.7 reaching a lower perplexity than sqrt [4].

Schedule-Free+. Without specifying the learning rate value or schedule shape, its actual learning rate curve rises to a peak under a constant nominal learning rate and then decays. This method outperforms the WSD baseline, reducing the time to reach the same loss by 31% in long-horizon settings [5].

effective learning rate of Schedule-Free+ compared with linear decay and WSD
1B model, 1000 tokens per parameter setting. Red is the actual lr of Schedule-Free+ (beta annealed from 0.8 to 0.965), black is linear decay, blue is WSD. Image source: Schedule-Free+.

Offline refined schedule. Constructed offline from the gradient norm sequence, it exhibits the same shape on eight tasks [6].

gradient norms and refined schedules across eight tasks
Results on eight tasks (ImageNet, IWSLT14, GPT, RoBERTa, DLRM, MRI, ViT, RCNN). Left column: gradient norm sequence; middle column: smoothed gradient norm; right column: refined schedule constructed from it, with inset on a logarithmic scale. Image source: Defazio et al. (2024), Figure 4.

minus-square-root. Hunyuan ELR proposes this based on the observed variation of AUS, reaching the target loss of 3.28 in 3,175 steps on Modded-nanoGPT Track 3 [1]. [2] uses a power-0.4 schedule on the same track and reaches the target in 3,150 steps. The two shapes are close, with a difference of 25 steps.

The four works produce the shape in different ways, yet the results all fall within the description of peak-then-decay. This phenomenon has causes at two levels.

4.1 Balance of bias and variance

The effect of a single-step update on the final model has two parts. The distance between the parameters and the initialization increases, corresponding to a decrease in bias. Gradient noise accumulates in the parameters, corresponding to an increase in variance. The learning rate determines the ratio of the two parts.

In the early phase of training, the bias term dominates, and a larger learning rate corresponds to faster bias reduction. In the late phase, the variance term dominates, and decaying the learning rate is equivalent to averaging over more updates, corresponding to a decrease in variance. The peak-then-decay shape is a combination of the requirements of these two phases.

The relative weight of bias and variance varies with task, model scale, and training budget. Therefore, the shapes that minimize $Bias+Variance$ form a set, which corresponds to an interval of $\alpha$ under the power-law parameterization. The existence of this set and its variation with conditions correspond to the two inferences in Section 5.

bias-variance plot for different cooldown shapes
Bias–variance distribution for each cooldown shape; the dashed line marks the position where $Bias+Variance$ reaches its minimum. Left: only lowered linear shapes compared, as the parameter decreases variance drops and bias rises; right: all nonlinear shapes and some lowered linear shapes, where sqrt and 0.7 fall near the dashed line, while square, cosine, mirror cosine, and linear lie above the line. The horizontal axis ranges differ between the left and right panels. Image source: Dremov et al. (2025), Figure 6.

The left panel shows the inverse relationship between bias and variance: as the parameter of lowered linear decreases, variance decreases and bias increases. The right panel shows the positions of each shape. sqrt and lowered linear 0.7 fall near the minimum line, while square, cosine, mirror cosine, and linear lie above the line. There is more than one shape near the minimum line [4]. The linear decay to zero reported in [7] also belongs to this category.

comparison of sqrt cooldown shape and lowered linear 0.7
Shape comparison of sqrt and lowered linear 0.7 within the cooldown segment (training progress 80%–100%). Image source: Dremov et al. (2025), Figure 7.

Two curves with different shapes perform comparably. One inference is that there is an upper bound on the tuning gain of the schedule shape: adjusting AdamW’s $\beta_2$ produces differences comparable to those from shape selection [4].

4.2 Conversion Involving Weight Norm in the Observed Quantity

Without the Hyperball constraint, part of the observed shape comes from the conversion in Section 1, and the rest comes from the schedule design. minus-square-root takes $\eta^\star$ as its design target, so the shape has to be written out explicitly. The Hyperball constraint removes the conversion, so $\eta_t$ itself must satisfy the shape requirement.

This conversion corresponds to an observation: the shape differences in the nominal learning rate are compressed after conversion, so the observed differences in the shape of $\eta^\star$ across settings are smaller than the shape differences in the nominal schedule. The AUS curves of the three nominal schedules in Section 1 nearly coincide after 1000 steps, which belongs to this category.

5. Summary

This article is organized around two questions. The first question is about the object of scheduling: the quantity being adjusted is $\eta_t$, and the quantity that takes effect is $\eta_t^\star$. The conversion factor between them is determined by the weight norm and changes during training. Hyperball fixes the weight norm, making the conversion factor constant, so its role is an implicit learning rate schedule. The second question is about shape: peak-then-decay is the solution to the bias–variance trade-off, where the bias term dominates early and the variance term dominates at the end.

The structure of the trade-off yields two inferences. The shapes that satisfy the balance form a set, so there is an upper bound on the tuning gain of shape selection. The relative weight of the two sides is determined by the training budget, model scale, and noise level, so the optimal shape differs across conditions.

Operational conclusions:

  1. When comparing different schedules, record the effective learning rate. Three nominal schedules with peak values differing by a factor of 2.4 can correspond to nearly identical AUS curves.
  2. When fitting scaling laws and transferring hyperparameters, align on $\eta^\star$ or on the cumulative angular displacement $\int\eta_t\mathrm{d}t$.
  3. Under the Hyperball constraint, the conversion has been removed, and the shape requirement must be satisfied by $\eta_t$, so a clear decay design is needed. There is an upper bound on the decay magnitude: when it is too large, late-training performance falls below that of MuonWD [2].
  4. The optimizable range of shape selection is limited; sqrt, lowered linear 0.7, and linear decay to zero perform comparably. Allocate the tuning budget first to the peak learning rate and $\beta_2$.

Two remaining issues. The quantitative correspondence between the shapes produced by adaptive methods and manually specified shapes has not been verified. The theoretical origin of the exponent in $\eta^*\propto T^{-0.32}$ has not been identified.

Related reading: the derivation in the width direction is in “On the Sphere: μP Scaling of Optimizers with the Hyperball Mechanism”, the estimation of the update matrix norm is in “Estimation of the Frobenius Norm of Update Matrices for Adam and Muon Optimizers”, and the schedule in the batch size direction is in “DASF: A Closed-Loop Batch Size Schedule-Free Method”.

References

[1] Tencent Hunyuan Pretrain Team (2026). From LR to ELR: A Better Heuristic for Pretraining Dynamics.

[2] Xiao, Y., Sun, J., Gao, Z., Wei, Z., Wang, C., Tao, R., Teng, J., & Dai, B. (2026). Hyperball May Not Be a Free Lunch. arXiv preprint arXiv:2607.22444.

[3] Ren, L., Liu, Y., Shen, Y., & Chen, W. (2026). Rethinking Language Model Scaling under Transferable Hypersphere Optimization. arXiv preprint arXiv:2603.28743.

[4] Dremov, A., Hägele, A., Kosson, A., & Jaggi, M. (2025). Training Dynamics of the Cooldown Stage in Warmup-Stable-Decay Learning Rate Scheduler. Transactions on Machine Learning Research (TMLR), 2025. arXiv preprint arXiv:2508.01483.

[5] Defazio, A. (2026). Schedule-Free+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models. arXiv preprint arXiv:2605.19095.

[6] Defazio, A., Cutkosky, A., Mehta, H., & Mishchenko, K. (2024). Optimal Linear Decay Learning Rate Schedules and Further Refinements. arXiv preprint arXiv:2310.07831.

[7] Bergsma, S., Dey, N., Gosal, G., Gray, G., Soboleva, D., & Hestness, J. (2025). Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs. ICLR 2025. arXiv preprint arXiv:2502.15938.

[8] Wen, K., Dang, X., Lyu, K., Ma, T., & Liang, P. (2026). Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization. arXiv preprint arXiv:2606.16899.

[9] Wan, R., Zhu, Z., Zhang, X., & Sun, J. (2020). Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGD. arXiv preprint arXiv:2006.08419.

Citation

If you need to cite this article, please refer to:

@article{zou2026hyperballimplicitschedule,
  title={Hyperball、effective lr 与峰值加衰减的形状},
  author={Zou, Jiaxuan},
  journal={Jiaxuan's Blog},
  year={2026},
  url={https://jiaxuanzou0714.github.io/blog/2026/hyperball-implicit-lr-schedule/}
}