Distillation

An agent does its job well, but it costs too much at volume or answers too slowly. How do we keep the quality and cut the cost? DistillationDistillation. Teaching a smaller, cheaper model to imitate a larger one on a narrow task once enough good examples exist. has a large model teach a small one. Quality stays roughly the same, and the small model is cheaper and faster. It has a second use too. It is the published repair for the forgetting that fine-tuning causes. A student can only be as good as its teacher. So distil only what the teacher already does well. This is one of the model-change options on the intelligence ladder.

How a model teaches a model

Supervised fine-tuning shows a model the right answer. Distillation shows it something richer. The student sees the teacher's odds for every possible next tokenToken. The unit AI usage is measured and billed in, roughly three-quarters of a word., not just the one it picked.

The student learns that "transferred" was chosen. It learns that "moved" was nearly chosen, and that "flew" was never in contention. Those relative preferences carry far more information per example than a bare answer. That is why a student can approach teacher quality with surprisingly modest training.

The modern refinement is on-policy distillation. The student does not just copy the teacher's transcripts. The student generates, and the teacher grades the student's own attempts. This teaches the student in the situations it actually gets into, including its own typical mistakes. Those differ from the situations the teacher would have been in. The published result is comparable quality at 9 to 30 times less training compute than scaling supervised fine-tuning.

The second use: repairing forgetting

Fine-tuning on your domain wears down general skill, and adapters do not prevent it. Adapters and supervised fine-tuning explains why.

The published repair is distillation with a specific teacher: the version of the same model from before the fine-tune. The fine-tuned student re-learns general behaviour from the model it used to be. It keeps the domain behaviour it just gained. This works because the two goals do not really conflict. The fine-tune simply moved weights it did not need to move. The original model is a perfect reference for where they belong.

So distillation is maintenance, not only compression. If you run adapters and fine-tuning or continued pretraining, plan to run this afterwards. Your general-skill eval tells you whether you need to.

foundation · comparison

One mechanism, two jobs

The student learns the teacher’s distribution, not just its answers. Who the teacher is decides which job you are doing.

Mixed evidence

The volume play

• Teacher: the big proven model
• Student: small, task-scoped
• Buys unit cost and latency
• Justified by call volume

The repair play

• Teacher: the pre-fine-tune model
• Student: your fine-tuned model
• Restores general capability
• Run after chapters 4 or 7

Diagram description: Two uses of distillation compared: compressing a proven capability into a smaller model, and repairing forgetting using the pre-fine-tune model as teacher. The comparison contains 2 groups: The volume play, containing Teacher: the big proven model, Student: small, task-scoped, Buys unit cost and latency, Justified by call volume; The repair play, containing Teacher: the pre-fine-tune model, Student: your fine-tuned model, Restores general capability, Run after chapters 4 or 7.

When it pays: the arithmetic

Distillation is a fixed cost that buys a lower cost per call. The decision is arithmetic:

• Volume. The saving is per call, so the fixed cost is spread over the calls. Divide the fixed cost by the saving on each call. That gives the number of calls it takes to pay the cost back. Compare that with your real call volume. No general threshold applies. The inputs are your own costs and volumes.
• Narrowness. A student holds teacher quality on a bounded task, not in general. The narrower the task, the smaller the student can be.
• A locked eval. The claim that the student holds quality is a measurement. It needs the evalEval (evaluation suite). A set of test tasks with known right answers, written by people who know the work, used to check an agent before and after any change.Read the page to exist before the student does. There is a tension here. The business case needs a fixed eval set. A live learning loop keeps changing the eval set. Decide which of the two you are running. The learning flywheel describes the live loop.

And one hard rule: never distil a capability the teacher does not reliably have. The student copies the teacher's errors as faithfully as its skills. It also lacks the size that sometimes rescued the teacher.

Routing work between larger and smaller models is costed in the observability and FinOps layer (R12). The training itself sits in the intelligence and learning layer (R06). A distilled model re-runs the evals at the change stage of the agent lifecycle.

Worked example: routine evidence extraction

Suppose a larger model does a bounded evidence-extraction task well enough at a utility. A smaller model may be worth testing for routine cases. That holds only if the volume and cost justify the training and upkeep.

Build examples with source references and checks. Hold out unusual documents, missing evidence and conflicting records. Compare the student with the original model and with a simpler extraction method. Send out-of-scope cases to the existing reviewed process.

Do not assume distillation repairs every problem caused by earlier training. Accept it only when evaluation shows the needed skills are kept and the full operating cost improves.

Where distillation fits

Distillation follows the other options rather than replacing them. It comes after adapters or continued pretraining as the forgetting repair. It can follow any of them to cut cost at volume. If what remains wrong is judgement on outcomes you can grade, that is reinforcement fine-tuning.

Evidence and limits

Mixed evidence The compute ratio and the forgetting repair are published work from October 2025. The cost arithmetic uses your own volumes, and no general number is claimed for it. The extraction example is a proposed design.

Next step

Read the next option, reinforcement fine-tuning.

Explore this topic

• On-policy distillation (October 2025), covered on intelligence and learning. It includes the repair with the pre-fine-tune model as teacher. It includes the 9 to 30 times training-compute comparison.
• Routing between larger and smaller models and the cost model behind it: observability and FinOps (R12).
• Industrial scenarios on this page are proposed designs, illustrated in the use cases.