# Distillation

How a large model teaches a small one, how it repairs forgetting, and the arithmetic that decides when it pays.

Author: Murali Sid (https://linkedin.com/in/muralisid)
Source: https://www.agenticarchitectureskills.com/ladder/distillation
Published: 2026-08-29
Updated: 2026-10-15
Licence: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/
Cite as: Murali Sid. Distillation. Agentic Architecture Skills. https://www.agenticarchitectureskills.com/ladder/distillation

---


An agent does its job well, but it costs too much at volume or answers too slowly. &#x2A;*How do we keep the quality and cut the cost?** <Term k="distillation">Distillation</Term> has a large model teach a small one. Quality stays roughly the same, and the small model is cheaper and faster. It has a second use too. It is the published repair for the forgetting that fine-tuning causes. A student can only be as good as its teacher. So distil only what the teacher already does well. This is one of the model-change options on [the intelligence ladder](/ladder).

## How a model teaches a model [#how-a-model-teaches-a-model]

Supervised fine-tuning shows a model the right answer. Distillation shows it something richer. The student sees the teacher's odds for every possible next <Term k="token">token</Term>, not just the one it picked.

The student learns that "transferred" was chosen. It learns that "moved" was nearly chosen, and that "flew" was never in contention. Those relative preferences carry far more information per example than a bare answer. That is why a student can approach teacher quality with surprisingly modest training.

The modern refinement is **on-policy distillation**. The student does not just copy the teacher's transcripts. The student generates, and the teacher grades the student's own attempts. This teaches the student in the situations it actually gets into, including its own typical mistakes. Those differ from the situations the teacher would have been in. The published result is comparable quality at **9 to 30 times less training compute** than scaling supervised fine-tuning.

## The second use: repairing forgetting [#the-second-use-repairing-forgetting]

Fine-tuning on your domain wears down general skill, and adapters do not prevent it. [Adapters and supervised fine-tuning](/ladder/adapters-and-fine-tuning) explains why.

The published repair is distillation with a specific teacher: &#x2A;*the version of the same model from before the fine-tune.** The fine-tuned student re-learns general behaviour from the model it used to be. It keeps the domain behaviour it just gained. This works because the two goals do not really conflict. The fine-tune simply moved weights it did not need to move. The original model is a perfect reference for where they belong.

So distillation is maintenance, not only compression. If you run adapters and fine-tuning or [continued pretraining](/ladder/continued-pretraining), plan to run this afterwards. Your general-skill eval tells you whether you need to.

<GuideFigure id="distillation-two-uses" />

<TeachingIllustration src="/images/ladder/distillation-mechanism-labeled-v4.webp" alt="A large teacher model passes a rich spread of possible outputs to a compact student. Below, a single hard answer carries less information." title="The distribution teaches more than the winning answer" caption="A teacher's relative confidence across alternatives carries information about boundaries and near misses. The student learns that shape, not merely a collection of final answers." />

## When it pays: the arithmetic [#when-it-pays-the-arithmetic]

Distillation is a fixed cost that buys a lower cost per call. The decision is arithmetic:

* **Volume.** The saving is per call, so the fixed cost is spread over the calls. Divide the fixed cost by the saving on each call. That gives the number of calls it takes to pay the cost back. Compare that with your real call volume. No general threshold applies. The inputs are your own costs and volumes.
* **Narrowness.** A student holds teacher quality on a bounded task, not in general. The narrower the task, the smaller the student can be.
* **A locked eval.** The claim that the student holds quality is a measurement. It needs the <Term k="eval">eval</Term> to exist before the student does. There is a tension here. The business case needs a fixed eval set. A live learning loop keeps changing the eval set. Decide which of the two you are running. [The learning flywheel](/architecture/learning-flywheel) describes the live loop.

And one hard rule: &#x2A;*never distil a capability the teacher does not reliably have.** The student copies the teacher's errors as faithfully as its skills. It also lacks the size that sometimes rescued the teacher.

Routing work between larger and smaller models is costed in [the observability and FinOps layer (R12)](/layers/r12-observability-and-finops). The training itself sits in [the intelligence and learning layer (R06)](/layers/r06-intelligence-and-learning). A distilled model re-runs the evals at the change stage of [the agent lifecycle](/architecture/agent-lifecycle).

## Worked example: routine evidence extraction [#worked-example-routine-evidence-extraction]

<TeachingIllustration src="/figures/ladder/industrial-distillation.svg" mobileSrc="/figures/ladder/industrial-distillation-mobile.svg" title="Test a smaller model for routine evidence work" alt="Four boxes in order. Teacher: a demonstrated capability. Examples: representative task outputs. Student: a smaller candidate model. Comparison: quality and full cost." caption="Proposed industrial example. Evaluate against the simpler existing process before adopting it." />

Suppose a larger model does a bounded evidence-extraction task well enough at a utility. A smaller model may be worth testing for routine cases. That holds only if the volume and cost justify the training and upkeep.

Build examples with source references and checks. Hold out unusual documents, missing evidence and conflicting records. Compare the student with the original model and with a simpler extraction method. Send out-of-scope cases to the existing reviewed process.

Do not assume distillation repairs every problem caused by earlier training. Accept it only when evaluation shows the needed skills are kept and the full operating cost improves.

## Where distillation fits [#where-distillation-fits]

Distillation follows the other options rather than replacing them. It comes after [adapters](/ladder/adapters-and-fine-tuning) or continued pretraining as the forgetting repair. It can follow any of them to cut cost at volume. If what remains wrong is judgement on outcomes you can grade, that is [reinforcement fine-tuning](/ladder/reinforcement-fine-tuning).

## Evidence and limits [#evidence-and-limits]

<EvidenceBadge status="mixed-evidence" /> The compute ratio and the forgetting repair are published work from October 2025. The cost arithmetic uses your own volumes, and no general number is claimed for it. The extraction example is a proposed design.

## Next step [#next-step]

Read the next option, [reinforcement fine-tuning](/ladder/reinforcement-fine-tuning).

<details>
  <summary>
    Explore this topic
  </summary>

  * On-policy distillation (October 2025), covered on [intelligence and learning](/layers/r06-intelligence-and-learning). It includes the repair with the pre-fine-tune model as teacher. It includes the 9 to 30 times training-compute comparison.
  * Routing between larger and smaller models and the cost model behind it: [observability and FinOps (R12)](/layers/r12-observability-and-finops).
  * Industrial scenarios on this page are proposed designs, illustrated in the [use cases](/use-cases).
</details>