AI Guides · 9 min read

What Is Model Distillation? How Teacher Models Train Smaller Ones

Model distillation trains a smaller student to mimic a larger teacher. See how it cuts inference cost, differs from fine-tuning, and what API terms restrict.

Illustration of a large luminous teacher form sending a soft fan of light toward a smaller student form

Model distillation is a training technique where a smaller student model learns to mimic a larger or more capable teacher model. By transferring the teacher’s output behavior into a compact architecture, teams cut inference cost and latency while retaining much of the teacher’s capability for specific production tasks.

What Is Model Distillation and How Does the Teacher-Student Setup Work?

Model distillation, also called knowledge distillation, is a training process that transfers behavior from a large, capable teacher model into a smaller or cheaper student model. Instead of training the compact model only on raw human-annotated examples, the team uses the teacher model to generate training signals that guide the student’s updates. The goal is a smaller model that runs faster and costs less per request while keeping much of the teacher’s behavior.

In a teacher-student model workflow, the teacher is already trained and stays frozen while producing targets on a dataset. The student updates its weights to match what the teacher produces. Teacher and student models do not need to share the same internal architecture. A team can pair a large, computationally heavy teacher with a much lighter student design suited for fast serving.

When practitioners talk about LLM distillation, the training signal shifts from simple class labels to language behavior. Distilling a language model often means training the student on text generated by the teacher, on preference labels ranking multiple responses, or on token-level probability distributions when those probabilities are available from the teacher. Later distillation variants also train the student to imitate intermediate features or reasoning traces from the teacher rather than relying solely on human-labeled datasets.

How Did Soft Targets and Dark Knowledge Start Model Distillation?

The modern formulation of knowledge distillation comes from the 2015 paper Distilling the Knowledge in a Neural Network by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Before that work, supervised classifiers were commonly trained on hard labels, where the correct class receives a score of one and every other class receives zero. That single-label target tells a model what the right answer is, while hiding how the teacher evaluates all the remaining options.

Hinton, Vinyals, and Dean showed that a trained teacher produces soft targets, meaning a full probability distribution across every possible class for each input. When an image classifier evaluates a photo of a pickup truck, it gives high probability to truck, a small non-zero probability to car, and virtually zero probability to carrot. Those small residual probabilities carry what the authors called “dark knowledge” about how classes relate to one another.

Training a student on soft targets passes along that structural similarity information, giving the smaller model far richer supervision per example than a one-hot label alone. The paper also introduces a temperature parameter that smooths the teacher’s output distribution during distillation so those secondary probabilities stand out more clearly during training. Because the student learns how the teacher distributes confidence across related answers, it can generalize better than a small model trained from scratch on hard labels alone.

Model Distillation vs. Fine-Tuning vs. Quantization

Model distillation, fine-tuning, and quantization are often grouped under model optimization, yet each method changes a model in a different way. Looking at model distillation vs. fine-tuning or weight compression helps clarify which tool solves which bottleneck.

Approach Primary Goal What Changes When to Use
Model distillation Transfer a large teacher’s behavior into a smaller, faster model A separate student model is trained to match the teacher’s outputs or probabilities You need lower latency or smaller hardware while preserving teacher-like behavior
Fine-tuning Adapt an existing model to a specific task, domain, or format Weights of the target model update on task data while size usually stays the same A base model already fits your latency budget and needs task specialization
Quantization Shrink memory usage and speed up weight arithmetic Numeric precision of existing weights drops, such as from 16-bit to 8-bit or 4-bit A model fits your accuracy needs and must fit into tighter VRAM or memory limits

Fine-tuning continues training an existing model on task-specific data so that model’s own behavior changes, typically without reducing its parameter count. Distillation has a different goal: it transfers behavior from a teacher into another, usually smaller, model. Quantization is not distillation either, because quantization reduces the numeric precision of an already-trained model’s weights rather than training a separate student to mimic a teacher.

Practitioners also encounter pruning, sparsity, mixture-of-experts (MoE) architectures, and retrieval-augmented generation (RAG) in the same conversations. Pruning, sparsity, and MoE designs modify compute or active parameter count inside a model, whereas RAG retrieves outside documents at inference time instead of compressing a teacher into a student. If you are still deciding whether to train, adapt, or swap sizes altogether, start with our guide on how to choose an AI model.

Why Do Teams Distill Language Models?

Engineering teams distill a language model primarily to cut inference cost and latency compared to running a large teacher model on every request. Running a flagship model across millions of routine classification, extraction, or drafting calls consumes heavy compute and stretches response times. Shifting that workload to a distilled student trained on the teacher’s behavior lets an application answer faster on smaller hardware or within tighter latency budgets.

Understanding what a token is in AI helps explain the billing side of that trade-off: every prompt and completion burns compute proportional to token volume and model size, so moving high-frequency traffic onto a compact student directly lowers per-request cost. Teams looking for broader ways to trim spend alongside smaller models can pair this with the tactics in our guide on how to reduce AI API costs.

In my own work reviewing production AI pipelines, teams rarely start with distillation on day one. Most start with a large teacher to validate product quality, then distill their own in-house teacher into a production student once traffic patterns stabilize. AI labs follow that same pattern when releasing smaller tiers of their own model families: distilling a model you built or properly licensed into a smaller student is a standard product-engineering practice.

What Do OpenAI Policies Say About Model Distillation?

Training a student model on outputs from a third-party hosted API is governed by provider contracts, and the summary below describes published policies as of November 16, 2026 (including a Usage Policies update marked effective November 20, 2026) rather than legal advice. Distilling a model you own, or a model whose license allows training on its outputs, is very different from pulling outputs out of a commercial API in violation of its terms.

Terms of Use and Usage Policies

Under OpenAI Terms of Use, users are prohibited from the following activity: “Use Output to develop models that compete with OpenAI.” That clause sits directly inside the restrictions governing how customers may use model outputs returned by OpenAI services.

The OpenAI Usage Policies page is marked effective November 20, 2026, so its updated training-and-developing-models section is a forthcoming rule not yet in effect as of November 16, 2026. On that page, the upcoming policy text states that users must not use OpenAI models or their outputs to generate, prepare, select, or supply training material or signals for models intended as alternatives or substitutes for OpenAI models or services, covering training data, tasks, rubrics, reinforcement learning environments, and training feedback such as labels, scores, rankings, preferences, rewards, or corrections. That forthcoming text also states that those restrictions do not prohibit uses expressly permitted by a customer’s agreement, including any Permitted Exception under OpenAI’s Services Agreement.

Enforcement Against Unauthorized Distillation Campaigns

In a public post titled Disrupting a coordinated model-distillation campaign, OpenAI described adversarial distillation as “the systematic and unauthorized use of one model’s outputs or reasoning to help train, reproduce, or improve another model.” OpenAI reported taking account and technical enforcement actions against a coordinated campaign that attempted to extract protected reasoning from its models.

Other hosted-model providers also restrict using API outputs to train competing models. Because exact permissions, enterprise exceptions, and open-weight licenses vary by vendor and tier, engineering teams should read the current terms of any hosted API or model license before collecting outputs for a training pipeline.

What Can Go Wrong With a Distilled Model?

A distilled student model can fail in production because copying a teacher’s outputs transfers blind spots alongside strengths. Even when a student looks convincing on a narrow benchmark, three recurring failure modes show up after deployment:

  • Inherited mistakes and biases: Because the student treats the teacher’s outputs as training truth, any factual errors, hallucinations, or systematic biases produced by the teacher can be learned and repeated by the student.
  • Out-of-distribution drops: Matching the teacher’s final answers on a curated training set does not guarantee matching the teacher’s reliability when real users send unfamiliar, noisy, or out-of-distribution inputs.
  • Coverage and safety gaps: Skills, edge cases, or safety behaviors that never appear in the distillation dataset can degrade noticeably in the student, leaving gaps that the original teacher handled safely.

Mitigating those failure modes requires careful dataset coverage and independent evaluation on real production edge cases rather than assuming the student inherited every property of the teacher.

FAQ

These frequently asked questions summarize how model distillation works, when teams use it, and what rules apply to hosted model outputs.

Is model distillation the same as fine-tuning?

Model distillation and fine-tuning serve different goals. Fine-tuning updates the weights of an existing model, usually keeping the same parameter count, so that model adapts to a specific task or style. Model distillation transfers behavior from a larger teacher model into a separate, typically smaller student model so teams can run inference faster and at lower cost.

Do teacher and student models need the same architecture?

Teacher and student models do not need to share the same architecture. Because distillation trains the student to match the teacher’s outputs, probability distributions, or intermediate signals, a smaller student network can learn from a larger teacher with a different internal structure. Teams typically pick a compact student architecture built for fast serving on their target hardware.

Can you distill a language model using OpenAI API outputs?

As of November 16, 2026, OpenAI’s Terms of Use (effective January 1, 2026) already prohibit using Output to develop models that compete with OpenAI. Separately, the OpenAI Usage Policies page adds a more detailed training-and-developing-models section marked effective November 20, 2026, which is not yet in force on November 16, 2026. Teams that want to build distilled models in compliance with provider terms rely on models they own or models licensed for output training.

How is model distillation different from quantization and RAG?

Model distillation, quantization, and retrieval-augmented generation address different bottlenecks. Model distillation trains a new, smaller student model to imitate a teacher model. Quantization reduces the numeric precision of an existing model’s weights, such as 16-bit to 8-bit or 4-bit, to shrink memory usage. Retrieval-augmented generation fetches outside documents during inference instead of compressing a teacher into a student.

What risks does a distilled student model carry?

A distilled student model can inherit any factual errors or biases present in the teacher model’s outputs. Matching a teacher’s answers on training examples also does not ensure reliability on out-of-distribution prompts. Finally, skills or safety behaviors omitted from the distillation dataset can weaken noticeably in the smaller student model during real-world deployment.

Sources

The technical definitions and provider policy details in this guide come from the primary research paper and official OpenAI documentation listed below.

Written by

Chunyu Cai

Chunyu Cai is the CEO of Lotook, LLC and an entrepreneur who writes about AI models, agents, and how they change everyday work. Chunyu holds a master's degree from Imperial College London.