Scaling Domain Data Repetition in LLM Pretraining
Mirrored from Hugging Face Daily Papers for archival readability. Support the source by reading on the original site.
Abstract
Under proportional scaling of model size and training tokens, optimal repetition of high-quality domain data increases mildly with scale and correlates with domain validation loss rather than unique data volume.
As large language models scale, their training-token budgets must also increase to maintain an appropriate tokens-per-parameter ratio (\(TPP\)). However, high-quality domain data is much harder to scale than general web data. As model size and the training-token budget increase, its fraction in the training mixture tends to decrease. Repeating the available high-quality data provides an effective way to counteract this dilution, but excessive repetition may lead to overfitting. We study this trade-off under practical LLM scaling, where the training-token budget grows proportionally with model size. For a fixed domain, we first find that, surprisingly at a fixed \(TPP\), the optimal repetition count mildly increases with model size. Across different domains, we find that the optimal repetition count is strongly negatively correlated with the final validation loss of a domain: domains with lower loss can generally benefit from more repetitions. In contrast, the amount of unique domain data is only weakly related to the optimal repetition count. These findings suggest that repetition counts tuned on smaller proxy models with the same \(TPP\) can provide a practical estimate for larger models.
Community
Models citing this paper
No model linking this paper
Datasets citing this paper
No dataset linking this paper
Spaces citing this paper
No Space linking this paper
Collections including this paper
More from Hugging Face Daily Papers
-
Luce: Relightable Gaussians for 3D Asset Generation
Aug 29
-
CritICL: Inference-Time Weak-to-Strong Generalization from Small Language Model Failure Modes
Aug 28
-
What Does an Evaluation License? A Commit-Bound Census of Claim-Relative Inference in Inspect Evals
Aug 28
-
EditaLive! Unified Character Video Editing for Live Streaming
Aug 28
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.