Pretraining and scaling as a methodology and scientific perspective

I research LLM pretraining and scaling. When I saw the GEN-1.5 loss curve spanning more than eight months of pretraining, my first reaction was:

This is a universal principle for turning energy into intelligence.

I see pretraining and scaling as a general method for developing reusable capabilities from experience through computation. Energy powers computation, data provides experience, and training encodes structure from that experience into weights. Language models, world models, and embodied AI can share this research approach.

1. General framework and scalability

What conditions does this approach require? Two points from the conversation between Yang Zhilin and Zhang Xiaojun stayed with me: [2]

• A sufficiently general framework that can learn from a broad range of experience.
• A scalable learning process that can continue to make use of more data and compute.

By framework, I mean the architecture, data representation, training objective, and optimization method. Once these are in place, I focus on the gains from additional compute, efficiency, stability, and predictability.

Dyna-2 scales human video pretraining to one million hours and achieves improved prediction on unseen robot data; after post-training with the same setup, real-world task performance also improves with data scale. Data composition and the video prediction objective also influence this result. [3]

The Dyna-2 results draw my attention to how pretraining improvements transfer to practical tasks. Different domains can use different architectures, metrics, and scaling curves, but each requires research into the learning conditions that support this transfer and continued improvement.

2. Designing learning conditions

When I first studied deep learning, I imagined new attention mechanisms based on long-term, short-term, and temporary memory, hoping to outperform standard attention. I would look for an explanation in human cognition, then use it to design a model.

Studying pretraining and scaling shifted my attention to another question: What architectures, data, and training objectives allow capabilities to develop through learning and improve with scale? Cognitive analogies can inspire hypotheses, but new mechanisms still need to justify their inclusion through learning outcomes, efficiency, and scalability. “Entities should not be multiplied without necessity” also applies to model design.

This is how I read Sutton’s The Bitter Lesson: prioritize general methods that can continue to benefit from more compute. The shift from feature engineering to feature learning leaves the specific form of features to training. Pretraining applies this approach to a broader range of data and tasks. [4]

Karpathy’s Software 2.0 further explains the researcher’s role: provide the architecture, data, and objective, then obtain a concrete implementation through optimization. This approach is especially valuable for tasks whose results we can evaluate but for which a complete program is difficult to write by hand. [5]

I therefore focus my research on the conditions for learning. Simple training objectives can produce complex representations and capabilities. Our understanding of the learning process may be sufficient to support capability improvements while our explanation of the model’s internal mechanisms remains incomplete. Building a capability can precede fully explaining it.

3. Pretraining and scaling

When studying these learning conditions, I focus on two questions: how to develop reusable capabilities, and how to keep improving them with more resources. Pretraining and scaling address these two questions, respectively. Pretraining learns from large datasets, encoding transferable structure into weights so that one model can adapt to many tasks. Scaling studies how increasing data and compute can continue to improve capabilities.

One aspect of pretraining I value is its ability to reduce the cost of subsequent learning. A pretrained model can adapt to new tasks through SFT or use demonstrations in context through ICL. GEN-1.5 makes this value tangible by demonstrating ICL from a single demonstration and task adaptation with a few gradient updates. [1]

When evaluating a pretrained model, I therefore also consider how efficiently it uses new experience: how many demonstrations, parameter updates, and how much compute it needs to learn a new task. An investment in pretraining can reduce learning costs across many subsequent tasks.

Scaling applies more broadly. Pretraining and RL can improve models with more training compute. At inference time, CoT, search, and environment interaction can also make use of additional compute. The gains from additional compute therefore need to be examined separately during training and inference.

4. Two milestones in training and inference

I see OpenAI’s 2020 scaling laws research and o1 in 2024 as important advances in training and inference, respectively.

In 2020, OpenAI systematically studied scaling laws for language models, enabling the relationship between parameter count, data size, compute, and loss to be used for prediction and resource allocation. This was a significant advance in training-time scaling; earlier empirical research already existed. [6][7]

In 2024, o1 demonstrated another important advance: models trained with reinforcement learning can effectively use more inference-time compute. Training can improve the way models use compute, and at inference time, additional compute is then applied to the current problem. [8]

CoT prompting was systematically proposed by Wei et al. in 2022, and research on test-time compute allocation existed before o1’s release. The significance of o1 lies in expanding the practical use of this capability through reinforcement learning. [9][10]

I also see the Seed Edge team’s EdgeBench as an important advance in test-time scaling. It allocates additional compute to long-horizon environment interaction: the agent continuously obtains observations and feedback, modifies plans, and verifies results, with some extended experiments exceeding 72 hours. The team observed a log-sigmoid relationship between the average score across tasks and interaction time. [11]

The experience here can be retained in context, external records, and modified plans, without updating weights. The connection that matters to me is that pretraining improves the ability to use new experience, and inference and environment interaction apply that ability to specific tasks.

In my view, these two milestones clarify two directions of scaling: training-time scaling improves model capability by increasing training compute; test-time scaling allocates more compute to inference, search, and environment interaction, improving results on the current task.

5. Efficiency and training stability

Once we establish where additional compute can improve results, we also need to study the cost of those improvements. In my own field of pretraining, efficiency determines the capability level achievable with a given budget, and training stability determines whether those gains can be realized over a complete training run.

I understand efficiency improvement as a leftward shift of the scaling curve: achieving the same loss requires fewer resources. When the horizontal axis is token count, it corresponds to token efficiency; when the horizontal axis is FLOPs, it corresponds to compute efficiency. Relating either measure to actual energy consumption also requires accounting for per-token compute cost and hardware efficiency.

By training stability, I mean controlling activation outliers and anomalous changes in gradient norms, weight norms, logits, and loss. At scales such as 3T or 10T parameters, I need to control these numerical and optimization issues to have confidence in scaling up further.

When scaling up, the Kimi K2 team encountered anomalous growth in attention logits. By using QK-Clip in MuonClip to adjust the Q and K projection weights, they completed pretraining on 15.5T tokens, reporting no loss spikes during the period. [12]

Stable training allows normal numerical growth and fluctuation. Decisions about intervention should be based on dynamics over the full training run and how these changes affect learning.

6. Predictability and detailed research

Efficiency and stability also need to be assessed before scaling up. I need to know whether the gains and training dynamics observed in small-scale experiments will hold at larger scales. This makes predictability another requirement for scaling research.

Scaling laws provide a basis for resource allocation. Prediction requires clarifying variables, metrics, and applicability: the power law of loss, task scores on logarithmic data axes, and log-sigmoid curves for long-horizon interaction have different meanings. [13]

I follow Marin because the team makes this process public. For the 535B-A23B run, they first use a scaling ladder to predict the full training trajectory, then check actual training against that prediction. Small-scale training can also provide references for dynamics such as gradient norm, helping to distinguish normal variation from deviations that require investigation. [14]

Delphi’s first experiment better illustrates why I care about this issue. The scaling law fit for small-scale experiments was good, but when scaled to $10^{22}$ FLOPs, the actual loss was 2.5% higher than predicted; training at $10^{23}$ FLOPs diverged. A good small-scale fit is still insufficient to guarantee that large-scale training matches predictions. [17]

Subsequently, the team adopted AdamH, added a learning rate correction that varies with training token count, and re-tuned hyperparameters. The improved $10^{23}$ FLOPs training completed, with a final loss about 0.2% higher than predicted. What I value is this process: correcting the training recipe based on failed results, then validating the predictability of scaling.

Marin previously discovered gradient growth issues through the scaling ladder and adopted logit z-loss; the public recipe also documents Hyperball-related methods like MuonH. Each method needs to be evaluated separately. [15]

Greg Yang et al.’s Tensor Programs V applies parameterization theory to scale-up: μP allows many optimal hyperparameters to remain stable as model width changes, and μTransfer transfers small-model hyperparameter tuning results to large models. [16]

This is also my understanding of scaling research: the general framework can be simple, but scale-up still requires detailed research. Parameterization, optimizer, data, and training dynamics jointly determine whether predictions hold at larger scales.

Make scaling more predictable and stable.

7. Summary

GEN-1.5, o1, and EdgeBench show me how general learning frameworks can use more experience and compute across domains and stages. The Marin experiments show that sustaining these gains also requires testing predictions, analyzing deviations, and revising training recipes.

Pretraining encodes broad experience into reusable capabilities and also improves subsequent learning efficiency. Scaling runs through both training and inference: training-time scaling improves model capabilities, while test-time scaling allocates more compute to specific tasks. Their implementations and curves may differ across domains, but the research methodology shares commonalities.

For me, the methodology is about designing conditions that enable continuous learning and scaling; the scientific perspective requires that the relationship between resource investment and capability improvement be predictable, testable, and correctable. Efficiency determines the returns on the same investment, stability ensures the learning process can continue, and predictability provides the basis for further scaling up investment. This is also my criterion for choosing research problems in pretraining and scaling.

References

[1] Generalist Team (2026). GEN-1.5: Embodied Foundation Models are One-Shot Learners. Generalist Blog.

[3] Dyna Robotics (2026). Dyna-2: A 1-Million-Hour Scaling Law for World-Action Models. Dyna Research.

[4] Sutton, R. S. (2019). The Bitter Lesson. Personal blog, PDF copy.

[5] Karpathy, A. (2017). Software 2.0. Medium.

[6] Kaplan, J., McCandlish, S., Henighan, T., et al. (2020). Scaling Laws for Neural Language Models. arXiv preprint arXiv:2001.08361.

[7] Hestness, J., Narang, S., Ardalani, N., et al. (2017). Deep Learning Scaling is Predictable, Empirically. arXiv preprint arXiv:1712.00409.

[8] OpenAI (2024). Learning to Reason with LLMs. OpenAI Blog.

[9] Wei, J., Wang, X., Schuurmans, D., et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv preprint arXiv:2201.11903.

[10] Snell, C., Lee, J., Xu, K., & Kumar, A. (2024). Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. arXiv preprint arXiv:2408.03314.

[11] ByteDance Seed (2026). EdgeBench: Measuring Real-World Environment Learning and Discovering a New Scaling Law. Seed Blog.

[12] Moonshot AI (2025). Kimi K2: Open Agentic Intelligence. Kimi Blog.

[13] Weng, L. (2026). Scaling Laws, Carefully. Lil’Log.

[14] Marin Community (2026). [Hero Run] 535B-A23B on 18T Tokens. GitHub Issue #8435.

[15] Marin Community. MoE Recipe. GitHub project documentation. Accessed: 2026-09-07.

[16] Yang, G., Hu, E. J., Babuschkin, I., et al. (2022). Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer. arXiv preprint arXiv:2203.03466.

[17] Held, W. (2026). Scaling Laws That Extrapolate 300× Past the Fit. Open Athena Blog.

[18] Dial, L. (2026). Improving our LLM Pretraining Efficiency. Open Athena Blog.

Citation

If you need to cite this article, please refer to:

@article{zou2026pretrainscaling,
  title={pretrain 和 scaling 作为一种方法论和科学观},
  author={Zou, Jiaxuan},
  journal={Jiaxuan's Blog},
  year={2026},
  url={https://jiaxuanzou0714.github.io/blog/2026/pretrain-scaling-methodology-scientific-perspective/}
}