Cap an LLM's training at fifth grade and its capability ceiling won't budge

Researchers built LittleLearner, a set of language models trained from scratch on an 88B-token corpus deliberately filtered to the U.S. K-5 curriculum. The corpus, distilled from FineWeb-Edu through a five-stage pipeline aligned to Common Core standards, strips out any concept, fact, or vocabulary taught above fifth grade. Models come in three sizes (0.6B, 1.3B, 5B), each paired with an unfiltered control that shares the same architecture, token budget, and training recipe. The goal is a controlled sandbox: because the knowledge boundary is known in advance, any behavior can be traced back to what the model was actually exposed to.

The headline result is that the pretraining filter acts as a hard ceiling on capability. Three standard interventions — scaling model size, post-training with GRPO (even on out-of-scope data), and in-context learning — all sharpen performance on material inside the K-5 boundary, but none meaningfully lifts performance beyond it. In the authors’ framing, these methods elicit skills the curriculum already taught rather than acquire genuinely new ones. That distinction is normally impossible to test, since production models are trained on everything at once, making it unclear whether a skill was learned or merely surfaced.

The setup opens up cleaner interpretability experiments: whether reinforcement learning can create capability from a restricted prior, how a model behaves at the edge of its knowledge (answer, abstain, or hallucinate), and how machine learners compare to children acquiring concepts like fractions or negative numbers. The 5B model is chattable in the browser, the checkpoints are released, and the work is documented in an arXiv paper.