What distillation leaves behind

A distilled model can land within a few points of a frontier model on reasoning benchmarks while running at a tenth of the parameter count. Garry Tan, who runs Y Combinator, has argued that US open-weight labs should be doing exactly what made DeepSeek’s R1 cheap to build: distilling large frontier models down into smaller, open, redistributable ones. He is right about the economics. What tends to get skipped is what distillation carries with it, and what it quietly leaves behind.

What distillation actually moves

Distillation is a teacher-student process. A large “teacher” model produces outputs - either the full probability distribution over next tokens, or just generated text and reasoning traces. A smaller “student” model is then trained to reproduce those outputs. The student ends up imitating the teacher’s behavior at a fraction of the size and inference cost.

DeepSeek did this in the open. It used outputs from its R1 model to fine-tune smaller Qwen and Llama base models, releasing distilled versions ranging from 1.5 billion up to 70 billion parameters. The larger distilled models approached R1’s performance on math and coding benchmarks while being far cheaper to run.

There are two flavors worth separating. In logit distillation, the student trains against the teacher’s full probability distribution, which requires access to the teacher’s internals - practical only when you own both models. In data distillation, the student trains on the teacher’s generated text alone, which is all you need if you can query the teacher through an API. The second kind is what makes distilling someone else’s model possible, and what makes it hard to police. You do not need the weights. You only need the outputs and a way to generate enough of them.

The important part is what moves and what does not. Distillation transfers behavior, not architecture, and not the process that produced the behavior. The student learns to copy answers. It does not learn why those answers were shaped the way they were. That separation - capability on one side, the training that governed it on the other - is the whole story for both security and safety.

Why Tan is pushing this now

The context is competitive. Chinese labs - DeepSeek, Alibaba’s Qwen - have shipped strong open-weight models cheaply and fast. Most US frontier labs keep their best models closed: OpenAI, Anthropic, Google. Meta’s Llama has carried most of the US open-weight banner, and that lead has narrowed.

Tan’s argument is straightforward: whoever’s open weights become the default - the thing developers download, fine-tune, and build products on - shapes an enormous amount of downstream behavior. If the US wants that default to be American, distillation is the cheapest path to a competitive open model.

The default-setting dynamic is real. Llama has been downloaded and fine-tuned into thousands of derivative models on Hugging Face. Each derivative inherits the base model’s assumptions about what is safe, what gets refused, and what a normal answer looks like. Whoever ships the base that everyone forks is writing defaults that propagate through an entire ecosystem, mostly invisibly. As industrial and geopolitical logic, that holds up. But a method about to be pushed as strategy deserves to have its properties named first.

The safety training does not come along

Safety in a frontier model is not baked into the raw weights of a pretrained network. It is layered on afterward: reinforcement learning from human feedback, refusal training, red-teaming, and output filtering. A large share of that work lives in the final fine-tuning stages - the part that teaches the model to say no.

When you distill from a teacher’s generated outputs, you copy its answers to the prompts you chose to ask. If your distillation set is full of math problems and coding tasks, you get a model good at math and coding. If that set never included the prompts where the teacher refused, the student never sees a refusal, and never learns one. Researchers have repeatedly shown that a few hundred fine-tuning examples can strip refusal behavior out of an aligned model. Distillation can produce the same result by omission - not by attacking the guardrails, just by never copying them.

The concrete failure looks like this: a distilled model that matches its teacher on writing code, with none of the teacher’s reluctance to write malware, because nobody distilled the “no” responses. You cannot assume a distilled open-weight model inherited any of its teacher’s safety properties. Those properties have to be measured on the student directly, from scratch.

Distillation as an attack surface

Two directions matter here.

The first is model extraction against closed APIs. Querying a paid frontier model at scale to harvest its outputs and train a competitor is a documented concern, not a hypothetical. It is simultaneously intellectual-property theft and API abuse, and providers actively watch for the query patterns that signal it. OpenAI has publicly pointed to suspected distillation of its models as a reason for tightening access. If distillation becomes standard practice, expect more aggressive rate limits, output watermarking, and terms-of-service enforcement around it.

The second is provenance. A distilled open-weight model is opaque. You receive weights - not the teacher’s identity, not the distillation data, not the prompts used. Backdoors and data poisoning can transfer through distillation. If the teacher was compromised, or the distillation set was seeded with trigger phrases, the student inherits the behavior, and you cannot grep weights for it. A distilled model is a supply-chain artifact shipped without a bill of materials. Every lesson the software industry learned about trusting dependencies with unknown provenance applies again, minus the ability to read the source.

What you cannot recall

Open weights are irreversible. When a closed model behaves badly, the provider can patch it, add a filter, or revoke access, and every user gets the fix. When open weights behave badly, there is no recall. Once a set of weights is downloaded and mirrored, it is permanent. Copies persist regardless of what the original lab does next.

That changes where the safety work has to happen. For a closed model, you can fix problems after release. For a distilled open-weight model, the release is the last moment of control. A model shipped with weak safety training is not a bug you patch later - it is a permanent artifact in the world. The stakes sit entirely before publication.

Testing the student, not the teacher

Re-testing a distilled model is not abstract work. Pull the same red-team prompt sets you would run against any deployment - harmful-instruction refusals, jailbreak resistance, prompt-injection handling, training-data extraction - and run them against the distilled weights directly. Compare refusal rates against the teacher’s published numbers, and treat any gap as real until proven otherwise. Where the teacher refused 98 percent of malware-generation prompts and the student refuses 40 percent, that is not measurement noise. That is the safety training that did not come along, measured.

What to actually do

If you build on distilled or open-weight models: treat the model as an untrusted dependency. Run your own safety and misuse evaluations before you ship. Do not assume inherited alignment. Test for the specific behaviors you care about - refusals, data leakage, prompt-injection resistance - on the exact model you are deploying, not on the teacher it came from.

If you distill and release: publish what you distilled from, on what data, and which safety evaluations you ran on the student. A model card should function like a software bill of materials. Re-run refusal and misuse tests on the distilled model specifically, because the teacher’s results say nothing about the student’s.

If you run an API that could be a distillation target: monitor for distillation-scale query patterns, rate-limit accordingly, and watermark outputs where the technique is viable. Assume your outputs are training data for someone else’s model.

And if you are setting policy or defending a network: treat “open-weight” and “safe” as independent variables. One does not imply the other. Distillation makes capability cheap to copy and makes safety expensive to preserve, and it does not preserve safety by default.

That asymmetry is the line to keep. Capability distills cheaply. Safety does not distill at all unless someone deliberately builds it back into the student, tests that it took, and does the work before the weights leave the building - because after that, there is no taking them back.