Data platform

The data platform layer (R02) holds company data for reporting and analysis: warehouses, lakehouses, plant data copies and dashboards. It is one of the fourteen layers. Agents read it before they answer or act. For agents it must provide four things. An agent finds only what the person asking may open. Every answer points back to the passage it came from. Numbers that matter come from fixed business definitions, or the agent says it does not know. Data is as fresh as the decision needs. A search index is a copy of the data, and it needs the same protection as the original.

What sits in this layer today

The shared vocabularies between these systems: CIM (IEC 61968 and 61970) in electricity, where used. Health uses SNOMED CT-AU, ICD-10-AM and LOINC. Energy and mining use ISA-95 object models, OSDU and ISO 14224. Vendor names are examples from industry knowledge (typical, unsourced). The full estates are on the utilities, healthcare and energy pages.

What agents add

Agents need company data turned into something they can safely answer from. That is called grounding: answering from trusted company sources. Treat grounding as a governed product, with an owner and rules. This layer feeds knowledge and memory, one of the seven capabilities an agent platform provides.

layer · architecture

Grounding as a governed product

Permissions crawl into index metadata and retrieval is pre-filtered on the caller's live identity; post-filtering is the anti-pattern.

1. 01System

   System

   Source systems

   Documents, records, events with native ACLs

2. 02Control

   Control

   Permission crawl

   ACLs into index metadata; ReBAC pre-computation

3. 03Evidence

   Evidence

   Governed index

   Version-stamped vectors; purpose-scoped where stakes demand

4. 04Control

   Control

   Pre-filtered retrieval

   Caller's live identity inside the query

5. 05Control

   Control

   Semantic contract

   Deterministic answers or refusal for high-stakes numerics

6. 06Agent

   Agent

   Agent context

   Answer with span-level source provenance

Embeddings are recoverable text; vectors inherit source classification and erasure obligations.

Diagram description: ACL-aware grounding pipeline from source systems through permission crawl into index metadata, pre-filtered retrieval on live identity, and span-level provenance, with fail-close branch. The map contains Source systems: Documents, records, events with native ACLs; Permission crawl: ACLs into index metadata; ReBAC pre-computation; Governed index: Version-stamped vectors; purpose-scoped where stakes demand; Pre-filtered retrieval: Caller's live identity inside the query; Semantic contract: Deterministic answers or refusal for high-stakes numerics; Agent context: Answer with span-level source provenance. Its connections are sources to crawl; crawl to index for fail-close on sync error; index to retrieval; retrieval to agent; semantics to agent for numeric questions. Important boundary: Embeddings are recoverable text; vectors inherit source classification and erasure obligations.

A permission check before the search

The index learns who may open each document. The source system's access list is copied into the index alongside each document. At query time the search is filtered on the caller's live identity, inside the query itself. Filtering the results after the search is the pattern to avoid. It leaks how many results exist, and it can leave too few to answer. The major search platforms support this natively. When the copy of permissions fails, the index denies access rather than allowing it.

An index protected like the data it copies

An index stores embeddings, which are numeric fingerprints of text. Those fingerprints can often be turned back into the text. One study rebuilt 92 percent of short inputs exactly from their embeddings. So vectors carry the source's classification, access rules, residency and privacy duties. Deleting a record deletes every copy made from it, vectors included. Anything built from several sources takes the strictest classification among them.

Fixed definitions for numbers that matter

For a high-stakes number, a set of fixed business definitions does one important thing. It turns a confidently wrong number into a right number or an honest refusal. These definitions are often kept in a semantic model. In one study, checking queries against such definitions raised accuracy from 54 to 72 percent. It also produced an "I do not know" on about 8 percent of questions. That refusal is the property that matters. Some teams reach similar accuracy through careful data modelling alone. Good modelling does the same job.

Storage sized to the number of documents

Start with the simplest storage that fits, and move up only when measurements say so. Below roughly 100,000 vectors, a plain search over all of them is enough. Postgres with the pgvector extension serves roughly 1 to 10 million vectors. Past about 5 million, it needs partitioning. Dedicated vector engines earn their place at very large scale, heavy write rates, or many tenants with heavy filters. Two cautions decide real moves. Building a large index takes tens of gigabytes of memory for hours. Filtered search can suddenly miss relevant items once a filter gets narrow.

Fresh enough for the decision

Change data capture streams each database change as it happens. Feeding it into an index gives freshness in minutes at close to batch cost. Full streaming commonly costs about ten times more than batch. One like-for-like case cost about $400 a month as a five-minute batch. The same job cost about $7,600 a month as a stream. So streaming is earned by how often decisions are made, as in fraud checks or operational triage. Similarity ranking ignores time. A superseded document can outrank its replacement unless freshness is part of the ranking.

Plan for re-embedding

Changing the embedding model means redoing the fingerprints for every document. Stamp every vector with its model version, and build the new index beside the old one before switching. The full discipline sits with agent data engineering (R14), which prepares data for agents.

The parts and where they run

What must be true before an agent touches it

The source permissions are right. No search tool fixes wrong permissions. Vendor estimates put about 16 percent of business-critical data in an overshared state. Find and fix oversharing before an agent amplifies it.

The search filters on the caller's live identity. The caller is the person who asked, carried through the delegation chain. The filter runs inside the query, never after it.

Four known failures are designed for. A revoked permission takes effect at the next copy, not at the click. Group membership drifts. Documents with no access list can default to public. Old oversharing gets amplified.

Every index carries its source's rules. Classification, residency and deletion travel with each vector and each answer.

Every answer shows its source. Each claim links back to the exact passage it came from.

High-stakes numbers go through fixed definitions, or the agent refuses. Watch how often it refuses. A sudden drop can mean it has started guessing.

The enforcement point is the index query. It runs in the search service, outside the model, on the caller's identity. This is enforcement outside the model. The path from source data to agent memory is set out in from data to memory.

Choices and their verdicts

Each entry is the common choice, tested against the simplest credible alternative. The full records are in the decision catalog.

• A semantic product or governed views: decide per use case, by how the answer fails, not by accuracy points.
• A dedicated vector database or the simplest store: move up one step at a time. Prefer the vector search built into the platform where the data already lives.
• Data mesh or a governed central store: use mesh ideas on a governed central store. Every shipped permission-copying design assumes a central index.
• Real-time or batch grounding: change data capture by default. Streaming is earned by how often decisions are made.
• One universal index or one per use case: both, in two levels. The detail is on the agent data engineering page.

Concerns this layer must honour

A concern is a requirement every layer must honour. This layer carries a specific duty for six of the ten.

• Identity and access. Search is filtered on the caller's live identity, and permission copy errors deny access.
• Traceability and audit. Every answer links to its passage. Every vector carries its model version.
• Grounding. This layer owns it, with agent data engineering. Fixed definitions answer or refuse.
• Sovereignty and residency. Indexes and embeddings keep the source's residency. Indexes can be scoped by region.
• Privacy. Embeddings count as personal data where the source does. Deletion reaches every vector and copy.
• Cost. Re-embedding, index size and freshness each have a price, set against how often decisions are made.

The full grid of concerns against layers is the concern matrix.

Evidence and limits

The embedding reconstruction result and the accuracy gain from checked definitions are peer reviewed or reproducible. The vector storage benchmarks and the semantic product comparisons come from vendors, and are marked [vendor] below. The oversharing estimate is a vendor figure, reported second-hand.

The storage thresholds are working figures from published benchmarks. They are not guarantees for your data. This guide recommends measuring on your own documents before any move.

Next step

See this layer at work in collecting and checking compliance evidence, where every answer must point to its source. Then read the next layer, the integration fabric (R03).

Explore this topic

• Data platform findings and sources.

Permissions and search

• Document-level directory access lists are native on the major search platforms.
• Connector permission copies ship with fail-closed behaviour: on an error, they deny.
• Relationship-based authorisation engines (the Zanzibar class) pre-compute allowed objects for the hard cases.
• About 16 percent of business-critical data is overshared, by vendor estimate, reported second-hand.
• One major vendor's fix is a scoped-search allowlist. Its own documentation calls it a stopgap.

Embeddings

• vec2text rebuilt 92 percent of short inputs exactly from their embeddings (EMNLP 2023, reproduced 2025).
• This result underpins the embedding risk named in the OWASP list for language model applications.

Business definitions

• Ontology-checked querying raised accuracy from 54 to 72 percent, with refusals on about 8 percent of questions.
• A vendor's paired test: 98 to 100 percent accuracy through definitions on the modelled scope [vendor].
• In the same test, text-to-SQL scored 84 to 90 percent [vendor].
• The definitions covered 72.7 percent of the full question set in that test [vendor].
• A semantic-document approach added 17 to 23 points across three frontier models [vendor].
• On the hardest public text-to-SQL benchmark, frontier models score from single digits to the low twenties, in percent.
• On that benchmark's predecessor, the same models score above 90 percent.

Storage

• pgvector at 1 million vectors answers in under 20 milliseconds, with above 95 percent recall.
• pgvector index builds get painful around 2 million vectors.
• Scale-out Postgres extensions reach roughly 50 million vectors.
• One comparison shows large latency and throughput gains for scale-out Postgres at 99 percent recall [vendor].
• At 100 million vectors with filters selecting 20 percent, Postgres p99 latency runs several times a dedicated engine's.
• Vector storage on object stores is generally available at up to 90 percent lower cost [vendor].
• Cheap object storage favours more, smaller, purpose-scoped indexes over one large one.

Freshness

• One documented requirement cost about $400 a month as a five-minute batch, and $7,600 as a stream.

Re-embedding

• A full re-embed takes hours on multi-GPU hardware, or four figures in API fees.
• Evaluate embedding models each quarter. Migrate only for test gains above a stated threshold. 5 percent is the working figure.
• Converting embeddings between models is published at roughly 100 times cheaper than re-embedding. Test it first.

Refusals and checks

• Refused: a widely circulated projection about agentic analytics projects failing by 2028. No primary source was found.
• Re-verify each quarter: vector search availability on the main platforms.
• Re-verify each quarter: adoption of the semantic interchange specification.
• Re-verify each quarter: embedding model releases that would trigger a re-embed.