# Continued pretraining

How a model learns a domain from raw text, what preparing that text involves, and where the budget should go.

Author: Murali Sid (https://linkedin.com/in/muralisid)
Source: https://www.agenticarchitectureskills.com/ladder/continued-pretraining
Published: 2026-08-29
Updated: 2026-10-15
Licence: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/
Cite as: Murali Sid. Continued pretraining. Agentic Architecture Skills. https://www.agenticarchitectureskills.com/ladder/continued-pretraining

---


A model still misreads your industry's documents, its terms and its notation. **How do we teach a model the language of our domain?** Continued pretraining teaches it from raw text, not from labelled examples. That is the material your organisation already has in bulk. The model keeps doing what pretraining always did: predicting the next word. Now it does so over your contracts, filings, tickets and manuals. It comes out reading your domain the way your people do. It suits an organisation with a large body of text. It is often confused with fine-tuning. Preparing the text is where the project succeeds or fails. This is one of the model-change options on [the intelligence ladder](/ladder).

## What the model learns from raw text [#what-the-model-learns-from-raw-text]

Predicting the next word sounds too simple to teach anything. It is worth seeing why it is not.

Take the phrase "the beneficial owner of the account is". To predict the next word, a model must know what beneficial ownership is. It must know what kinds of entities own accounts. It must know what tends to follow in this kind of document.

Prediction quality stands in for understanding. Improving it over billions of tokens of your material forces the model to absorb your domain's concepts, relationships and conventions. Nobody labels anything. The text itself is the teacher. Every next word is a small test with a known answer.

There is also a mechanical reason general models read specialist text badly. It shows before any learning happens. Models read in <Term k="token">tokens</Term>. A token is a word or part of a word. A tokeniser trained on general text splits unfamiliar words into fragments. A drug name, a legal term or a SWIFT message type becomes four or five meaningless pieces. SWIFT is the messaging standard banks use for payments. The model spends effort putting back together words your industry treats as single units. Continued pretraining teaches the model how your vocabulary behaves, even through awkward splitting. That is part of why its fluency in the domain improves so visibly.

## How this differs from fine-tuning [#how-this-differs-from-fine-tuning]

|             | Supervised <Term k="fine-tuning">fine-tuning</Term> | Continued pretraining                                           |
| ----------- | --------------------------------------------------- | --------------------------------------------------------------- |
| Input       | Pairs: a question and the wanted answer             | Raw text, as it already exists                                  |
| Volume      | Thousands of examples                               | Billions of tokens                                              |
| Teaches     | How to behave                                       | What the domain is                                              |
| Preparation | High: someone writes the answers                    | Different: nobody writes anything, but the text must be curated |

The preparation row needs more detail. "You already have the data" is true, and also misleading.

<GuideFigure id="sft-vs-cpt" />

<TeachingIllustration src="/images/ladder/continued-pretraining-mechanism-labeled-v2.webp" alt="A huge stream of raw domain text is curated and filtered. General material is mixed in. The mix repeatedly reshapes the model as it learns the domain. A small supervised set then teaches the task. Domain first, task second." title="Domain first, task second" caption="Continued pretraining changes how the model represents the domain by learning from raw material at enormous scale. A much smaller supervised set then teaches the particular job the domain-aware model must perform." />

## Preparing the text is the project [#preparing-the-text-is-the-project]

A model absorbs whatever the text contains, without judgement. The work is deciding what that should be:

* **Removing duplicates.** Company archives repeat themselves: templates, boilerplate, forwarded chains and near-identical versions. Repetition gives extra weight to whatever repeats.
* **Filtering for quality.** Errors from scanned pages, corrupted exports and machine-generated logs teach noise.
* **Removing what must not be kept.** Take out anything you must not retain, and anything you would not want generated back. Examples are passwords in old tickets, personal data outside its retention basis, and privileged material.
* **The rights question.** Training on the text requires the right to do so, and obligations follow the weights. A model trained on your text is a derived artifact of it, much like an index or a copy. Classification, residency and erasure duties reach the resulting model. [The concern matrix](/architecture/concern-matrix) sets out that inheritance. It is sharper here than for any other option, because the model is large, expensive and cannot forget selectively. Those duties are governed in [the governance, risk and sovereignty layer (R11)](/layers/r11-governance-risk-sovereignty). The text pipeline belongs to [the agent data engineering layer (R14)](/layers/r14-agent-data-engineering).
* **The mix.** Training only on domain text sharpens the domain and dulls everything else. In practice, domain text is mixed with general text to hold general skill. The forgetting repair on [distillation](/ladder/distillation) applies here as it does to fine-tuning.

## Where the budget goes [#where-the-budget-goes]

<TeachingIllustration src="/images/ladder/continued-pretraining-corpus-labeled-v2.webp" alt="A disordered archive passes through filters for rights, duplication, contamination, quality and balance. A sealed evaluation collection stays outside training." title="The corpus is the product" caption="Rights, provenance, deduplication, contamination control, quality and balance determine what the model learns. The evaluation set must remain sealed outside that production line or the final score stops meaning anything." />

One study modelled the split directly. It took a fixed budget of 30 billion tokens and split it between continued pretraining and supervised fine-tuning. It used models with 7 billion parameters across health, chemistry and coding. In every domain, the best split put **roughly 99.99 percent of the budget into continued pretraining**. The small remainder went to fine-tuning. It reported an 11.6 percent improvement over a leading domain-adapted comparison. It used up to 95 percent less budget (OpenReview, 8 October 2025).

Read it as a direction, not a recipe. It is one study, three domains and one model size. The lasting conclusions are the order and the proportion. Learning the domain is the big job, done on raw text. Learning the task is the small job on top, done on pairs. Some teams invert this. They hand-write tens of thousands of pairs to inject knowledge. That is the expensive job done with the wrong tool.

## Worked example: industrial language at a utility [#worked-example-industrial-language-at-a-utility]

<TeachingIllustration src="/figures/ladder/industrial-continued-pretraining.svg" mobileSrc="/figures/ladder/industrial-continued-pretraining-mobile.svg" title="Evaluate domain adaptation for industrial language" alt="Four boxes in order. Corpus: permitted technical material. Adaptation: learn domain regularities. Task tuning: shape useful behaviour. Evaluation: check domain and general tasks." caption="Proposed industrial example. Evaluate against the simpler existing process before adopting it." />

A utility or mining company may have technical language, notation and document layouts that a general model handles poorly. Domain training is a candidate only after a fair comparison with retrieval, a glossary, examples and a different model.

Review the training material's rights, sensitivity, quality and intended use. Keep changing asset facts and requirements available as governed evidence. Training a model on a document does not replace citing the applicable version during a task.

Measure the domain improvement, any loss of general skill, the preparation cost and the ongoing update burden. Do not assume training will shrink every prompt or improve every later workflow.

## Where continued pretraining stops [#where-continued-pretraining-stops]

For almost every enterprise, continued pretraining plus fine-tuning is as far as model change needs to go. [Custom pretraining and architecture](/ladder/custom-pretraining) is for one specific situation: data that is not text at all.

A model trained this way re-runs the evals at the change stage of [the agent lifecycle](/architecture/agent-lifecycle). [The learning flywheel](/architecture/learning-flywheel) sets out how it is tested and promoted. The training itself sits in [the intelligence and learning layer (R06)](/layers/r06-intelligence-and-learning).

## Evidence and limits [#evidence-and-limits]

<EvidenceBadge status="preprint-or-prototype" /> The budget finding is a single study at 7B scale. It is presented as a direction, not a recipe. The way tokenisers split words is standard and well understood. The utility example is a proposed design.

## Next step [#next-step]

Read the last option, [custom pretraining and architecture](/ladder/custom-pretraining).

<details>
  <summary>
    Explore this topic
  </summary>

  * Modelling Optimal Trade-Off Between Continued Pre-Training and Supervised Fine-Tuning for LLM Domain Adaptation, OpenReview, 8 October 2025: [https://openreview.net/forum?id=guUUlHPXRw](https://openreview.net/forum?id=guUUlHPXRw)
  * Derived artifacts inheriting obligations: [the concern matrix](/architecture/concern-matrix) and [agent data engineering](/layers/r14-agent-data-engineering).
  * The forgetting repair: [distillation](/ladder/distillation), and [intelligence and learning](/layers/r06-intelligence-and-learning).
  * Industrial scenarios on this page are proposed designs, illustrated in the [use cases](/use-cases).
</details>