Proxy-KD: Distilling GPT-4-Class Models Without Peeking Inside Them

Knowledge distillation normally relies on access to a teacher model’s internal states—logits, hidden activations—to train a smaller student. That breaks down with proprietary models like GPT-4, which expose only their text outputs. This paper proposes Proxy-KD, which inserts an intermediate proxy model to bridge that gap, letting a small model absorb knowledge from a black-box teacher despite the missing internals.

The reported results are notable: Proxy-KD doesn’t just outperform other black-box distillation approaches, it beats traditional white-box techniques that do have full access to the teacher’s internals. If that holds up, the implication is that the proxy acts as a richer signal source than raw output text alone, recovering much of what closed APIs deliberately withhold.

For practitioners, the takeaway is a more viable path to building capable small models on top of API-only frontier systems—relevant to cost reduction, on-device deployment, and reducing dependence on per-call API pricing. It also sharpens an ongoing tension for model providers, since effective black-box distillation makes proprietary capabilities easier to replicate from outputs alone.