# Custom pretraining and architecture

Two meanings of 'our own model': a foundation model built from scratch, or a different model for non-text data.

Author: Murali Sid (https://linkedin.com/in/muralisid)
Source: https://www.agenticarchitectureskills.com/ladder/custom-pretraining
Published: 2026-08-29
Updated: 2026-10-15
Licence: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/
Cite as: Murali Sid. Custom pretraining and architecture. Agentic Architecture Skills. https://www.agenticarchitectureskills.com/ladder/custom-pretraining

---


A vendor or an internal team proposes building "our own model". **Do we need to build a model of our own?** Almost never. Two very different things are sold under similar words. One is training your own foundation model from scratch. A foundation model is a large general model trained on huge amounts of text. That is a question of economics, and it is almost never the answer. The other is designing a different kind of model because your data is not documents. It may be networks, sequences or signals. That is a real discipline with a recognisable use. Every other option on [the intelligence ladder](/ladder) works around a model someone else built. This one builds models. That is a different profession.

## Path one: a foundation model of your own [#path-one-a-foundation-model-of-your-own]

### What it is [#what-it-is]

Pretraining from scratch starts with random weights. The model trains on trillions of tokens. Then it is tuned into something usable. Everything the model knows must come from the text you assembled. There is no base model to start from.

### What it costs [#what-it-costs]

It costs what a frontier laboratory spends. Computing costs run to tens of millions and more. It needs a standing research team. It takes time to reach anything competitive, and the field moves on meanwhile. The result also goes stale. Open and commercial models improve on someone else's budget. Yours improves only on yours.

### When it is genuine [#when-it-is-genuine]

One case is a sovereignty rule that forbids outside model weights, for a nation or a systemically important regulated firm. Sovereignty rules are covered in [the governance, risk and sovereignty layer (R11)](/layers/r11-governance-risk-sovereignty). Another case is a body of text so large and so unusual that general models are wrong for it. Even then you need the money and the team. If you are reading this page to find out whether you qualify, you do not.

### The question that ends most proposals [#the-question-that-ends-most-proposals]

Ask what [continued pretraining](/ladder/continued-pretraining) on the same text would achieve. It delivers most of what people want from "our own model". It uses the same data, at a small fraction of the cost. And someone else maintains the base model and its upgrades.

### One published result [#one-published-result]

If you do take this path, the timing of domain data matters. One study mixed 0.1 to 10 percent domain data into pretraining from scratch. That beat the standard route of general pretraining followed by adaptation, across three domains. It cut the tokens needed for a given domain performance by up to 1.75 times \[vendor]. The study used models of 300M to 3B parameters and a 200B-token budget. It was published by a company that sells data curation. It informs how to run path one, not whether to take it.

<GuideFigure id="top-of-market-paths" />

<TeachingIllustration src="/images/ladder/custom-pretraining-paths-labeled-v3.webp" alt="Two paths. Path 1 is your own foundation model. Random weights, vast data and vast compute make a model. It has extreme cost and goes stale fast. It is only for real sovereignty needs. Path 2 is a different architecture. Network-shaped data goes into a graph model. It produces a risk score and an explanatory subgraph. The agent receives them through a tool. A banner asks first whether continued pretraining would solve this." title="Two expensive-sounding paths solve different problems" caption="Training from random weights is a decision about sovereignty and economics at frontier scale. Choosing a different architecture depends on the shape of the data. A graph model calculates over relationships, and the agent receives its result as governed evidence." />

## Path two: when your data is not text [#path-two-when-your-data-is-not-text]

A language model is a machine for sequences of tokens. Some company data is not like that. No amount of prompting, retrieval or fine-tuning changes that. Examples are transaction networks, supply chains, sensor streams and molecular structures. For these, the model itself is different. The most common choice is a graph neural network. The engineering around it is different too.

### How a graph model works [#how-a-graph-model-works]

Each point in the network, called a node, starts with a vector describing it. In each round, every node updates its vector using its neighbours' vectors. After one round, a node knows about its neighbours. After two, it knows about their neighbours, and so on outward. The model then predicts from the node vectors. Is this account likely part of a fraud ring? Is this component about to fail? The structure itself, who connects to whom, is what the model learns from. A document model cannot see that structure.

### What goes wrong [#what-goes-wrong]

Graph models have well-known failure modes. Two matter most. Stack too many rounds and every node in a connected region ends up with the same vector. This is called over-smoothing. It is why production graph models are often only two or three rounds deep. Information from far away must squeeze through a few links into vectors of fixed size. This is called over-squashing. The two pull against each other. Adding rounds to ease over-squashing makes over-smoothing worse.

Someone who has really worked here has had to resolve that trade in the design itself. Common fixes are rewiring the graph, attention, or pooling in stages, not a bigger setting. Large graphs bring more problems. They must be cut or sampled to fit in memory, and every cut severs real relationships. Training spread across many machines also behaves differently from the textbook. The full list is in the research block at the end of this page.

### When it is genuine [#when-it-is-genuine-1]

The thing you predict is a property of a structure, not of a document. The scale is large enough for these failure modes to bite. That situation is easy to recognise, and it has little to do with agents.

## Worked example: a specialist model behind an agent [#worked-example-a-specialist-model-behind-an-agent]

<TeachingIllustration src="/figures/ladder/industrial-custom-pretraining.svg" mobileSrc="/figures/ladder/industrial-custom-pretraining-mobile.svg" title="Connect a specialist model to the agent" alt="Four boxes in order. Problem: a defined prediction task. Baseline: rules or an existing model. Specialist: train only if justified. Tool result: evidence for the agent." caption="Proposed industrial example. Evaluate against the simpler existing process before adopting it." />

At a utility or a mine, a specialist model could estimate a production quantity. It could describe a sequence of equipment readings, or analyse relationships in a network. The agent uses that model through a tool. It does not become the model.

Start with the measurement or prediction task and a credible reference outcome. Compare conventional features, existing models and simpler statistical methods before building a new architecture. Include the data, research, deployment and maintenance effort.

Keep the specialist model's units, uncertainty, calibration and conditions of use with its result. An agent's fluent explanation must not turn an uncertain estimate into an official production record. See [production measurement](/use-cases/production-measurement).

## How this connects to the rest [#how-this-connects-to-the-rest]

Every other option on the intelligence ladder works around a fixed model. This page is about building models. The two need different budgets, different staff and a different risk lifecycle. They meet at a tool boundary: the agent calls the specialist model as a tool.

A new specialist model is tested and promoted like any other model change, on [the learning flywheel](/architecture/learning-flywheel). Models and their training sit in [the intelligence and learning layer (R06)](/layers/r06-intelligence-and-learning).

One habit carries across every option. Diagnose the structural failure before spending money. Never assume a bigger version of what you have will fix it.

## Evidence and limits [#evidence-and-limits]

<EvidenceBadge status="mixed-evidence" /> The pretraining-mix result is vendor-published research at small model scale, from 18 March 2026 \[vendor]. The graph failure modes are standard and well documented. They are not specific to this guide. The worked example is a proposed design.

## Next step [#next-step]

See how any change to a model is tested and promoted on the learning flywheel. Or return to [the Start page](/start).

<details>
  <summary>
    Explore this topic
  </summary>

  * The Finetuner's Fallacy, DatologyAI, 18 March 2026 \[vendor]. Domain mix at 0.1 to 10 percent, models of 300M to 3B, a 200B-token budget. [https://www.datologyai.com/blog/finetuners-fallacy](https://www.datologyai.com/blog/finetuners-fallacy)
  * The alternative for almost all readers: [continued pretraining](/ladder/continued-pretraining).
  * The matched-control discipline: [how to test a context design](/patterns/how-to-test-a-context-design).
  * Industrial scenarios on this page are proposed designs, illustrated in the [use cases](/use-cases).

  Graph model failure modes, for practitioners. These are standard in the published literature on graph neural networks.

  * **Over-smoothing.** Each round averages neighbours. Stack enough rounds and every node in a connected region converges to the same vector. Depth then destroys the distinctions the model exists to draw.
  * **Over-squashing.** Information from a node several hops away passes through a neighbourhood that grows exponentially. It is compressed into fixed-size vectors, so long-range signal is crushed at bottleneck edges.
  * **The two pull against each other.** Rounds added to relieve over-squashing worsen over-smoothing. Real fixes are structural: rewiring, attention or hierarchical pooling.
  * **Representation and gradient collapse.** Learned vectors stop being distinct. The learning signal dies with depth, so parts of the network silently stop training.
  * **Partitioning and sampling.** A graph with billions of edges does not fit in accelerator memory. Unlike documents, it cannot be cut freely, because every cut severs real relationships. Partitioning into clusters loses the edges between clusters. Sampling neighbourhoods adds variance and bias to what each node sees.
  * **Distributed convergence.** Spread across many machines, training stops behaving like the single-machine textbook. Giant effective batches change the loss landscape. Stale gradients arrive from busy workers. Communication cost starts to dictate the algorithm.
</details>