Continued pretraining

A model still misreads your industry's documents, its terms and its notation. How do we teach a model the language of our domain? Continued pretraining teaches it from raw text, not from labelled examples. That is the material your organisation already has in bulk. The model keeps doing what pretraining always did: predicting the next word. Now it does so over your contracts, filings, tickets and manuals. It comes out reading your domain the way your people do. It suits an organisation with a large body of text. It is often confused with fine-tuning. Preparing the text is where the project succeeds or fails. This is one of the model-change options on the intelligence ladder.

What the model learns from raw text

Predicting the next word sounds too simple to teach anything. It is worth seeing why it is not.

Take the phrase "the beneficial owner of the account is". To predict the next word, a model must know what beneficial ownership is. It must know what kinds of entities own accounts. It must know what tends to follow in this kind of document.

Prediction quality stands in for understanding. Improving it over billions of tokens of your material forces the model to absorb your domain's concepts, relationships and conventions. Nobody labels anything. The text itself is the teacher. Every next word is a small test with a known answer.

There is also a mechanical reason general models read specialist text badly. It shows before any learning happens. Models read in tokensToken. The unit AI usage is measured and billed in, roughly three-quarters of a word.. A token is a word or part of a word. A tokeniser trained on general text splits unfamiliar words into fragments. A drug name, a legal term or a SWIFT message type becomes four or five meaningless pieces. SWIFT is the messaging standard banks use for payments. The model spends effort putting back together words your industry treats as single units. Continued pretraining teaches the model how your vocabulary behaves, even through awkward splitting. That is part of why its fluency in the domain improves so visibly.

How this differs from fine-tuning

The preparation row needs more detail. "You already have the data" is true, and also misleading.

foundation · comparison

Teaching behaviour, teaching the domain

Different data, different volume, different job. Most enterprises have the raw text and few labelled pairs.

Preprint or prototype

Supervised fine-tuning

• Input: question and answer pairs
• Thousands of examples
• Teaches how to behave
• Someone must write the answers

Continued pretraining

• Input: raw text as it exists
• Billions of tokens
• Teaches what the domain is
• The corpus must be curated

Together

• Pretrain first, then fine-tune
• Domain first, task second
• Most budget to pretraining
• Repair forgetting after

Diagram description: Supervised fine-tuning and continued pretraining compared: pairs versus raw text, thousands versus billions of tokens, behaviour versus knowledge. The comparison contains 3 groups: Supervised fine-tuning, containing Input: question and answer pairs, Thousands of examples, Teaches how to behave, Someone must write the answers; Continued pretraining, containing Input: raw text as it exists, Billions of tokens, Teaches what the domain is, The corpus must be curated; Together, containing Pretrain first, then fine-tune, Domain first, task second, Most budget to pretraining, Repair forgetting after.

Preparing the text is the project

A model absorbs whatever the text contains, without judgement. The work is deciding what that should be:

• Removing duplicates. Company archives repeat themselves: templates, boilerplate, forwarded chains and near-identical versions. Repetition gives extra weight to whatever repeats.
• Filtering for quality. Errors from scanned pages, corrupted exports and machine-generated logs teach noise.
• Removing what must not be kept. Take out anything you must not retain, and anything you would not want generated back. Examples are passwords in old tickets, personal data outside its retention basis, and privileged material.
• The rights question. Training on the text requires the right to do so, and obligations follow the weights. A model trained on your text is a derived artifact of it, much like an index or a copy. Classification, residency and erasure duties reach the resulting model. The concern matrix sets out that inheritance. It is sharper here than for any other option, because the model is large, expensive and cannot forget selectively. Those duties are governed in the governance, risk and sovereignty layer (R11). The text pipeline belongs to the agent data engineering layer (R14).
• The mix. Training only on domain text sharpens the domain and dulls everything else. In practice, domain text is mixed with general text to hold general skill. The forgetting repair on distillation applies here as it does to fine-tuning.

Where the budget goes

One study modelled the split directly. It took a fixed budget of 30 billion tokens and split it between continued pretraining and supervised fine-tuning. It used models with 7 billion parameters across health, chemistry and coding. In every domain, the best split put roughly 99.99 percent of the budget into continued pretraining. The small remainder went to fine-tuning. It reported an 11.6 percent improvement over a leading domain-adapted comparison. It used up to 95 percent less budget (OpenReview, 8 October 2025).

Read it as a direction, not a recipe. It is one study, three domains and one model size. The lasting conclusions are the order and the proportion. Learning the domain is the big job, done on raw text. Learning the task is the small job on top, done on pairs. Some teams invert this. They hand-write tens of thousands of pairs to inject knowledge. That is the expensive job done with the wrong tool.

Worked example: industrial language at a utility

A utility or mining company may have technical language, notation and document layouts that a general model handles poorly. Domain training is a candidate only after a fair comparison with retrieval, a glossary, examples and a different model.

Review the training material's rights, sensitivity, quality and intended use. Keep changing asset facts and requirements available as governed evidence. Training a model on a document does not replace citing the applicable version during a task.

Measure the domain improvement, any loss of general skill, the preparation cost and the ongoing update burden. Do not assume training will shrink every prompt or improve every later workflow.

Where continued pretraining stops

For almost every enterprise, continued pretraining plus fine-tuning is as far as model change needs to go. Custom pretraining and architecture is for one specific situation: data that is not text at all.

A model trained this way re-runs the evals at the change stage of the agent lifecycle. The learning flywheel sets out how it is tested and promoted. The training itself sits in the intelligence and learning layer (R06).

Evidence and limits

Preprint or prototype The budget finding is a single study at 7B scale. It is presented as a direction, not a recipe. The way tokenisers split words is standard and well understood. The utility example is a proposed design.

Next step

Read the last option, custom pretraining and architecture.

Explore this topic

• Modelling Optimal Trade-Off Between Continued Pre-Training and Supervised Fine-Tuning for LLM Domain Adaptation, OpenReview, 8 October 2025: https://openreview.net/forum?id=guUUlHPXRw
• Derived artifacts inheriting obligations: the concern matrix and agent data engineering.
• The forgetting repair: distillation, and intelligence and learning.
• Industrial scenarios on this page are proposed designs, illustrated in the use cases.