Distillation makes fast followers, never frontier leaders

Garry Tan’s pitch to US open-weight labs is short: stop trying to out-train the frontier from scratch, and distill from existing frontier models the way DeepSeek did to close the gap with Chinese open-weight releases like Qwen and Kimi. The straight answer is that distillation is a real, proven technique - not a trick - but the version that is actually cheap and effective is far narrower than the headline suggests. It works when the teacher is a model you are permitted to use as a teacher, when the task is something you can verify, and when you accept that the student inherits a hard ceiling set by whatever it learned from.

Strip the idea down and distillation is capability transfer, not capability creation. A large “teacher” model produces outputs, and a smaller “student” model is trained to reproduce them. The savings are genuine: you can build a capable small model for a tiny fraction of what it costs to pretrain a frontier system, because you are skipping the expensive part - discovering the capability in the first place - and copying the result. What you get for that price is a fast follower, not a leader. The student can match a slice of the teacher’s behavior cheaply. It cannot invent anything the teacher did not already contain.

The part of Tan’s argument that decides whether it holds up is the one usually left out: where the teacher’s outputs come from, and whether you are legally allowed to train on them. That single constraint controls everything downstream - cost, quality, and whether the resulting model is something a US lab can actually ship. Before anyone talks pipelines or benchmarks, that is the question that determines if the strategy is viable or just a demo that works until legal reads the terms of service.

Mechanically, there are two very different operations hiding under the phrase “distill frontier models,” and they are not interchangeable. The first is logit-level, or soft-label, distillation: the student trains to match the teacher’s full probability distribution over the next token, not just the token it picked. This transfers the most information per example, but it requires access to the teacher’s internals - its logits - which means you need the actual weights. You can only do this with an open-weight teacher or a model you trained yourself. The second is sequence-level, or synthetic-data, distillation: you take the teacher’s finished completions, treat them as training data, and fine-tune the student on them like any supervised set. This is what most people mean, and it is the only form you can run through a closed API, because all you need is the text coming out.

DeepSeek is the reference case because it made the pipeline legible. DeepSeek-R1 generated long chain-of-thought reasoning traces, filtered those traces for correctness, and fine-tuned smaller dense models - Qwen and Llama bases - on the surviving data. The leverage was never a secret model or a clever prompt. It was the loop: generate at scale, verify against ground truth, throw away everything that failed, train on what remained. That is orchestration. The intelligence lived in the filtering step, not in the generation step, and that is the detail that gets lost when people describe distillation as “copying a big model into a small one.”

Seen as a system, distillation is a data pipeline with a probabilistic generator at the front and a deterministic validation layer in the middle. The model is the smallest component. The real work is generation cost, deduplication, correctness filtering, and building a curriculum the student can actually learn from. This is why distillation is strong in domains where outputs are cheap to verify - math you can check, code you can execute, structured tasks with ground truth - and why it degrades in domains where you cannot verify anything, like open-ended writing or judgment calls. In those cases you are copying the teacher’s style and its mistakes with equal fidelity, and no filter can tell the difference. The technique is only as good as your ability to score what comes out of the teacher.

The first place people get this wrong is the assumption that you can legally distill the closed frontier models - that a US open-weight lab can quietly pull completions from GPT, Claude, or Gemini and train on them. The terms of service for those providers prohibit using their outputs to train competing models, and a lab that intends to publish weights is exactly the competitor those clauses were written to stop. So the cheapest, most attractive route is contractually closed to anyone serious about shipping. The teachers you can legally distill from are open-weight models: Llama, Qwen, DeepSeek itself. Which produces an awkward result for the strategic version of Tan’s argument - “US labs should distill frontier models” often resolves, in practice, to “US labs should distill from Chinese open weights,” because that is where the legally usable strong teachers are. That cuts directly against the point about competing with them.

The second mistake is treating distillation as a free lunch where the student can somehow exceed the teacher. It cannot, at least not on the distribution it was trained to copy. The student can match a narrow band of the teacher’s behavior at lower cost, but it inherits the teacher’s blind spots, its biases, and its failure modes, and it loses the long tail of capability that made the teacher valuable in the first place. Distillation compresses; it does not discover. What you ship is a model that behaves like the teacher inside the distribution you sampled and falls off a cliff outside it - often without any warning sign until a real workflow hits the edge you never generated data for.

The third and most consequential mistake is believing distillation replaces pretraining and closes the frontier gap on its own. It does not. Someone still has to train the frontier teacher, and that is the expensive, uncertain, capital-heavy work that distillation is explicitly avoiding. If every open-weight lab distills and no one funds frontier pretraining, the whole ecosystem stalls at the current ceiling, because there is nothing new to copy from. What genuinely changed - and what makes Tan’s argument land now rather than two years ago - is that reasoning models made distillation far more effective, because reasoning traces can be verified. Correctness filtering turned distillation from lossy compression into capability transfer with real quality control. That shift is the substance behind the pitch. It is also its limit: a fast-follow strategy is downstream of frontier investment, not a substitute for it, and confusing the two is how a lab ends up permanently one step behind the models it is copying.

The version that survives contact with production starts by fixing the two variables that decide everything, and refusing to begin the run until both are settled: a legally usable teacher and a verifiable task. In practice that means an open-weight teacher whose license actually permits training derivatives - a Llama, Qwen, or DeepSeek checkpoint you have read the license on, not skimmed the marketing for - and a target domain where you can score an output without a human in the loop. Everything downstream inherits those two choices. Get them wrong and no amount of pipeline engineering saves the run, because you either cannot ship what you build or you cannot tell good outputs from bad ones.

The architecture is a data pipeline, not a training script, and treating it as a training script is the first way teams waste money. Front end: the teacher generates many candidate completions per prompt through rejection sampling - higher temperature, multiple samples, deliberate over-generation. Middle, and this is where the leverage lives: a deterministic verifier scores every candidate. Unit tests executed in a sandbox for code, a numeric or symbolic checker for math, schema validation for structured extraction, reference-answer matching wherever you have ground truth. You keep only what passes, deduplicate near-identical traces, and check the survivors for contamination against your evaluation set. Back end: a straightforward supervised fine-tune of the student on the surviving traces, then evaluation on held-out problems the teacher never generated against. The probabilistic generator is wrapped in deterministic control on both sides. That is the entire mechanism, and the model is the smallest part of it.

The trade-offs are real and worth naming before you commit budget. Generation is the dominant cost, not training - you will discard somewhere between 60 and 90 percent of what the teacher produces, and you pay compute for all of it, including the failures. You need a verifier per domain, and building a reliable verifier is frequently harder than running the distillation it feeds. LLM-as-judge is a fallback only when you have a tight, explicit rubric, and even then you are now copying the judge’s biases into the student alongside everything else. And you scope narrow on purpose. A student that is excellent at Python generation and useless at open-ended prose is a shippable, defensible outcome. A student that is mediocre across everything is not, and undirected distillation produces the second one by default.

Make it concrete. A ten-person US lab wants a 7B model that is strong at competitive-programming-style code reasoning, and it intends to publish the weights. It picks an open-weight reasoning teacher with a permissive license - a DeepSeek-R1-grade distillation checkpoint or a license-checked Qwen-Coder variant - and not GPT, Claude, or Gemini, because shipping open weights is exactly the use the closed providers’ terms of service prohibit. Teacher settled, task settled, and only now does the engineering start.

The run itself: assemble roughly 50,000 programming problems that come with hidden test suites. For each problem, sample eight to sixteen solutions from the teacher with full chain-of-thought. Execute every solution against its tests inside an isolated sandbox with hard timeouts and resource caps, because you are running untrusted generated code at volume. Keep only the traces that pass all tests, deduplicate by problem and solution structure, and reserve a clean, contamination-filtered slice for evaluation. From millions of generated attempts you might keep 200,000 to 400,000 verified traces. Fine-tune the 7B base on those. The bill is dominated by generation and sandbox execution rather than the training step, and it lands at a small fraction of what pretraining a comparable model from scratch would cost. This is the case that makes Tan’s pitch look obviously correct.

Now watch it break, exactly where the theory predicts. The same lab tries to extend the model to code review and architecture advice - judgment tasks with no executable oracle - and the pipeline has nothing to filter on. Output quality collapses to whatever the teacher said, mistakes included, because the correctness gate that made the code-reasoning distillation work does not exist for unverifiable tasks. The model demos fine and falls off the moment a real workflow hits the edge you could not score. The lesson is the one from earlier stated in reverse: the intelligence was always in the filter, and a task with no filter has no distillation story at all, only expensive imitation.

Tan is right that distillation is the correct move for a small open-weight lab, and wrong if the takeaway is that it closes the frontier gap. It is a fast-follow strategy - high leverage, low cost, genuinely effective - inside the narrow band of tasks you can verify and from teachers you are legally clear to train on. Push it outside that band and it is a demo that works until legal or a real user finds the edge.

The strategic problem is the one that keeps getting skipped: the legally usable strong teachers are disproportionately Chinese open weights. “US labs should distill frontier models” resolves, in practice, to “US labs should distill from Qwen and DeepSeek.” That is a sound tactical move and a strange answer to a competitiveness argument, because a fast follower is by definition downstream of whoever trained the teacher. If the strongest legally usable teachers are foreign open weights, distillation makes you a faster follower of them, not a competitor to them.

So do it, and scope it honestly. Build the verification-gated pipeline, ship the narrow verifiable win, and treat the resulting model as precisely what it is: a cheap, capable copy of a slice of someone else’s system. Do not let a distillation program become the reason to stop funding the expensive, uncertain work of training a teacher of your own. The day every open-weight lab distills and no one pretrains is the day the ecosystem freezes at the current ceiling, because there is nothing new left to copy. Distillation buys you speed. It does not buy you the frontier, and confusing the two is how a lab ends up permanently one step behind the models it is copying.

---

Contains a referral link.