What happens when an LLM never sees material beyond fifth grade?<br>The 5B LittleLearner model trained from scratch on LittleCurriculum, a corpus restricted to K–5 material, answers this question and allows to test the effect of post-training, prompting and scaling on moving beyond the pretraining knowledge boundary.</p>\n","updatedAt":"2026-08-17T07:02:32.519Z","author":{"_id":"67c02f5cd067f2e27edae514","avatarUrl":"/avatars/f3bb665f7aa38bae0ca660ebe292ef29.svg","fullname":"Jana Zeller","name":"Jana-Z","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8498846888542175},"editors":["Jana-Z"],"editorAvatarUrls":["/avatars/f3bb665f7aa38bae0ca660ebe292ef29.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.13545","authors":[{"_id":"6a7f2efcf747ea94019af4ff","name":"Fanfei Li","hidden":false},{"_id":"6a7f2efcf747ea94019af500","name":"Jana Zeller","hidden":false},{"_id":"6a7f2efcf747ea94019af501","name":"Manuel Prada-Corral","hidden":false},{"_id":"6a7f2efcf747ea94019af502","name":"Thaddäus Wiedemer","hidden":false},{"_id":"6a7f2efcf747ea94019af503","name":"Prasanna Mayilvahanan","hidden":false},{"_id":"6a7f2efcf747ea94019af504","name":"Ryan Cotterell","hidden":false},{"_id":"6a7f2efcf747ea94019af505","name":"Wieland Brendel","hidden":false}],"mediaUrls":["https://cdn-uploads.huggingface.co/production/uploads/67c02f5cd067f2e27edae514/2WOolcKKz8Yeleovru6vM.jpeg"],"publishedAt":"2026-08-13T00:00:00.000Z","submittedOnDailyAt":"2026-08-17T00:00:00.000Z","title":"LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure","submittedOnDailyBy":{"_id":"67c02f5cd067f2e27edae514","avatarUrl":"/avatars/f3bb665f7aa38bae0ca660ebe292ef29.svg","isPro":false,"fullname":"Jana Zeller","user":"Jana-Z","type":"user","name":"Jana-Z"},"summary":"Modern language models are trained on heterogeneous web-scale text corpora. Consequently, studying knowledge and skill acquisition is difficult, as prior exposure to related content is hard to characterize. To address this challenge, we introduce LITTLECURRICULUM, a curated 88B-token pretraining corpus tailored to U.S. elementary school material, explicitly excluding concepts, facts, and vocabulary taught above Grade 5. Training a 5B-parameter LLM from scratch on LITTLECURRICULUM yields LITTLELEARNER, a model with sufficient language competence for open-ended evaluation, yet with clear knowledge and capability boundaries mapped to interpretable curriculum guidelines. We release LITTLECURRICULUM and LITTLELEARNER as a developmentally restricted sandbox to study how models acquire, represent, and use data under a well-defined training scope. We illustrate the sandbox's utility in a first suite of experiments on injecting new knowledge through post-training and in-context learning. These methods let LITTLELEARNER better utilize existing knowledge, but do not raise out-of-scope capabilities. Our findings underscore the value of this controlled environment for future investigations.","upvotes":3,"discussionId":"6a7f2efdf747ea94019af506","projectPage":"https://littlelearner-ll.github.io/","ai_summary":"A curated elementary-grade pretraining corpus and 5B-parameter model create a controlled sandbox for studying knowledge acquisition, representation, and bounded capability growth via post-training and in-context learning.","ai_keywords":["pretraining corpus","LLM","in-context learning","post-training","knowledge injection","curriculum-guided training"],"ai_summary_model":"thinkingmachines/Inkling-Small","organization":{"_id":"68f2384a5e53d7d6240bd063","name":"ELLIS-Institute-Tuebingen","fullname":"ELLIS Institute Tübingen","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/63d86dbf3130cadcaf8bdd11/yXv5xW2lR52xL8s_hKWC6.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"67c02f5cd067f2e27edae514","avatarUrl":"/avatars/f3bb665f7aa38bae0ca660ebe292ef29.svg","isPro":false,"fullname":"Jana Zeller","user":"Jana-Z","type":"user"},{"_id":"68f5532dbe52f6c6e87f0994","avatarUrl":"/avatars/96434a9e536573645701a13934a7b410.svg","isPro":false,"fullname":"Fanfei Li","user":"ffli","type":"user"},{"_id":"647c4a2692182942d7c2e698","avatarUrl":"/avatars/bcddf5fe49aa092a2645f70812108348.svg","isPro":false,"fullname":"HWANCHANG","user":"HwanChang0106","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"68f2384a5e53d7d6240bd063","name":"ELLIS-Institute-Tuebingen","fullname":"ELLIS Institute Tübingen","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/63d86dbf3130cadcaf8bdd11/yXv5xW2lR52xL8s_hKWC6.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.13545.md","query":{}}">
LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure
Abstract
A curated elementary-grade pretraining corpus and 5B-parameter model create a controlled sandbox for studying knowledge acquisition, representation, and bounded capability growth via post-training and in-context learning.
Modern language models are trained on heterogeneous web-scale text corpora. Consequently, studying knowledge and skill acquisition is difficult, as prior exposure to related content is hard to characterize. To address this challenge, we introduce LITTLECURRICULUM, a curated 88B-token pretraining corpus tailored to U.S. elementary school material, explicitly excluding concepts, facts, and vocabulary taught above Grade 5. Training a 5B-parameter LLM from scratch on LITTLECURRICULUM yields LITTLELEARNER, a model with sufficient language competence for open-ended evaluation, yet with clear knowledge and capability boundaries mapped to interpretable curriculum guidelines. We release LITTLECURRICULUM and LITTLELEARNER as a developmentally restricted sandbox to study how models acquire, represent, and use data under a well-defined training scope. We illustrate the sandbox's utility in a first suite of experiments on injecting new knowledge through post-training and in-context learning. These methods let LITTLELEARNER better utilize existing knowledge, but do not raise out-of-scope capabilities. Our findings underscore the value of this controlled environment for future investigations.
Community
What happens when an LLM never sees material beyond fifth grade?
The 5B LittleLearner model trained from scratch on LittleCurriculum, a corpus restricted to K–5 material, answers this question and allows to test the effect of post-training, prompting and scaling on moving beyond the pretraining knowledge boundary.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2608.13545 in a model README.md to link it from this page.
Cite arxiv.org/abs/2608.13545 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2608.13545 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.