# Data platform
How agents get company data they may see and can trust: permission checks before the search, an index protected like the data it copies, fixed definitions for numbers that matter, and freshness matched to the decision.
Author: Murali Sid (https://linkedin.com/in/muralisid)
Source: https://www.agenticarchitectureskills.com/layers/r02-data-platform
Published: 2026-08-20
Updated: 2026-10-15
Licence: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/
Cite as: Murali Sid. Data platform. Agentic Architecture Skills. https://www.agenticarchitectureskills.com/layers/r02-data-platform
---
The data platform layer (R02) holds company data for reporting and analysis: warehouses, lakehouses, plant data copies and dashboards. It is one of the [fourteen layers](/layers). Agents read it before they answer or act. For agents it must provide four things. An agent finds only what the person asking may open. Every answer points back to the passage it came from. Numbers that matter come from fixed business definitions, or the agent says it does not know. Data is as fresh as the decision needs. A search index is a copy of the data, and it needs the same protection as the original.
## What sits in this layer today [#what-sits-in-this-layer-today]
| System category | Electricity distributor | Water utility | Hospital network | Generator, oil and gas, or miner |
| -------------------------- | ----------------------------------------------------------------------------------------- | ---------------------------------- | ------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| Warehouse or lakehouse | Snowflake, Databricks, Microsoft Fabric, SAP BW or Datasphere | Same as the distributor | Enterprise data warehouse (SQL Server, Oracle). Newer builds on Snowflake, Databricks, Microsoft Fabric | Lakehouse (Databricks, Snowflake, Microsoft Fabric) |
| Business intelligence (BI) | Power BI, Tableau, SAP Analytics Cloud | Same as the distributor | Power BI, Qlik, Tableau | Power BI, Tableau |
| Copies of operational data | A copy of the plant historian, the archive of control-system readings | Same, plus hydraulic model outputs | Clinical data repository (InterSystems HealthShare, Orion Health) | Process historian (AVEVA PI System, AspenTech IP.21, Honeywell PHD). Industrial data platform (Cognite Data Fusion, AVEVA CONNECT) |
| Specialist data sets | Interval meter data, copied from meter data management | Water-quality lab data | Clinical registries. Linked data for population health | Lab results (LabWare, Thermo SampleManager). Subsurface data (OSDU). Geology and block models (acQuire, Maptek) |
| Data built for regulators | Data marts for the Australian Energy Regulator's templates | Reports to the state regulator | State data collections | Emissions reporting (NGER). Reserve statements for miners (JORC) |
| How fresh it is | Loaded overnight, so good for history and poor for the current state (typical, unsourced) | Same as the distributor | Nightly batch extracts from the patient administration system and medical record | The historian collects plant readings continuously |
The shared vocabularies between these systems: CIM (IEC 61968 and 61970) in electricity, where used. Health uses SNOMED CT-AU, ICD-10-AM and LOINC. Energy and mining use ISA-95 object models, OSDU and ISO 14224. Vendor names are examples from industry knowledge (typical, unsourced). The full estates are on the [utilities](/use-cases/utilities), [healthcare](/use-cases/healthcare) and [energy](/use-cases/energy) pages.
## What agents add [#what-agents-add]
Agents need company data turned into something they can safely answer from. That is called grounding: answering from trusted company sources. Treat grounding as a governed product, with an owner and rules. This layer feeds [knowledge and memory](/memory), one of [the seven capabilities an agent platform provides](/architecture/system-view).
### A permission check before the search [#a-permission-check-before-the-search]
The index learns who may open each document. The source system's access list is copied into the index alongside each document. At query time the search is filtered on the caller's live identity, inside the query itself. Filtering the results after the search is the pattern to avoid. It leaks how many results exist, and it can leave too few to answer. The major search platforms support this natively. When the copy of permissions fails, the index denies access rather than allowing it.
### An index protected like the data it copies [#an-index-protected-like-the-data-it-copies]
An index stores embeddings, which are numeric fingerprints of text. Those fingerprints can often be turned back into the text. One study rebuilt 92 percent of short inputs exactly from their embeddings. So vectors carry the source's classification, access rules, residency and privacy duties. Deleting a record deletes every copy made from it, vectors included. Anything built from several sources takes the strictest classification among them.
### Fixed definitions for numbers that matter [#fixed-definitions-for-numbers-that-matter]
For a high-stakes number, a set of fixed business definitions does one important thing. It turns a confidently wrong number into a right number or an honest refusal. These definitions are often kept in a semantic model. In one study, checking queries against such definitions raised accuracy from 54 to 72 percent. It also produced an "I do not know" on about 8 percent of questions. That refusal is the property that matters. Some teams reach similar accuracy through careful data modelling alone. Good modelling does the same job.
### Storage sized to the number of documents [#storage-sized-to-the-number-of-documents]
Start with the simplest storage that fits, and move up only when measurements say so. Below roughly 100,000 vectors, a plain search over all of them is enough. Postgres with the pgvector extension serves roughly 1 to 10 million vectors. Past about 5 million, it needs partitioning. Dedicated vector engines earn their place at very large scale, heavy write rates, or many tenants with heavy filters. Two cautions decide real moves. Building a large index takes tens of gigabytes of memory for hours. Filtered search can suddenly miss relevant items once a filter gets narrow.
### Fresh enough for the decision [#fresh-enough-for-the-decision]
Change data capture streams each database change as it happens. Feeding it into an index gives freshness in minutes at close to batch cost. Full streaming commonly costs about ten times more than batch. One like-for-like case cost about $400 a month as a five-minute batch. The same job cost about $7,600 a month as a stream. So streaming is earned by how often decisions are made, as in fraud checks or operational triage. Similarity ranking ignores time. A superseded document can outrank its replacement unless freshness is part of the ranking.
### Plan for re-embedding [#plan-for-re-embedding]
Changing the embedding model means redoing the fingerprints for every document. Stamp every vector with its model version, and build the new index beside the old one before switching. The full discipline sits with [agent data engineering (R14)](/layers/r14-agent-data-engineering), which prepares data for agents.
### The parts and where they run [#the-parts-and-where-they-run]
| Part | What it does | Where it runs |
| -------------------- | -------------------------------------------------------------------- | -------------------------------------- |
| Permission copier | Copies each source's access list into the index, and denies on error | Beside each connector |
| Governed index | Serves search with permissions inside the query | Your existing search or vector store |
| Business definitions | Answer high-stakes numbers the same way every time, or refuse | Warehouse or semantic model |
| Change capture | Keeps indexes as fresh as the decision needs | Your existing streaming or batch tools |
| Catalog and lineage | Supply owners, definitions and data history to agents | Your existing catalog |
| Source links | Tie every claim to the passage it came from | Stored with the index |
## What must be true before an agent touches it [#what-must-be-true-before-an-agent-touches-it]
**The source permissions are right.** No search tool fixes wrong permissions. Vendor estimates put about 16 percent of business-critical data in an overshared state. Find and fix oversharing before an agent amplifies it.
**The search filters on the caller's live identity.** The caller is the person who asked, carried through the [delegation chain](/architecture/delegation-chain). The filter runs inside the query, never after it.
**Four known failures are designed for.** A revoked permission takes effect at the next copy, not at the click. Group membership drifts. Documents with no access list can default to public. Old oversharing gets amplified.
**Every index carries its source's rules.** Classification, residency and deletion travel with each vector and each answer.
**Every answer shows its source.** Each claim links back to the exact passage it came from.
**High-stakes numbers go through fixed definitions, or the agent refuses.** Watch how often it refuses. A sudden drop can mean it has started guessing.
The enforcement point is the index query. It runs in the search service, outside the model, on the caller's identity. This is [enforcement outside the model](/architecture/enforcement). The path from source data to agent memory is set out in [from data to memory](/architecture/data-to-memory).
## Choices and their verdicts [#choices-and-their-verdicts]
Each entry is the common choice, tested against the simplest credible alternative. The full records are in the [decision catalog](/decisions).
* [A semantic product or governed views](/decisions#cd-8): decide per use case, by how the answer fails, not by accuracy points.
* [A dedicated vector database or the simplest store](/decisions#cd-9): move up one step at a time. Prefer the vector search built into the platform where the data already lives.
* [Data mesh or a governed central store](/decisions#cd-10): use mesh ideas on a governed central store. Every shipped permission-copying design assumes a central index.
* [Real-time or batch grounding](/decisions#cd-12): change data capture by default. Streaming is earned by how often decisions are made.
* [One universal index or one per use case](/decisions#cd-11): both, in two levels. The detail is on the agent data engineering page.
## Concerns this layer must honour [#concerns-this-layer-must-honour]
A concern is a requirement every layer must honour. This layer carries a specific duty for six of the ten.
* **Identity and access.** Search is filtered on the caller's live identity, and permission copy errors deny access.
* **Traceability and audit.** Every answer links to its passage. Every vector carries its model version.
* **Grounding.** This layer owns it, with agent data engineering. Fixed definitions answer or refuse.
* **Sovereignty and residency.** Indexes and embeddings keep the source's residency. Indexes can be scoped by region.
* **Privacy.** Embeddings count as personal data where the source does. Deletion reaches every vector and copy.
* **Cost.** Re-embedding, index size and freshness each have a price, set against how often decisions are made.
The full grid of concerns against layers is the [concern matrix](/architecture/concern-matrix).
## Evidence and limits [#evidence-and-limits]
The embedding reconstruction result and the accuracy gain from checked definitions are peer reviewed or reproducible. The vector storage benchmarks and the semantic product comparisons come from vendors, and are marked \[vendor] below. The oversharing estimate is a vendor figure, reported second-hand.
The storage thresholds are working figures from published benchmarks. They are not guarantees for your data. This guide recommends measuring on your own documents before any move.
## Next step [#next-step]
See this layer at work in [collecting and checking compliance evidence](/use-cases/compliance-evidence), where every answer must point to its source. Then read the next layer, the [integration fabric (R03)](/layers/r03-integration-fabric).
Explore this topic
* [Data platform findings](/library/layers/r02-data-platform/findings) and [sources](/library/layers/r02-data-platform/sources).
**Permissions and search**
* Document-level directory access lists are native on the major search platforms.
* Connector permission copies ship with fail-closed behaviour: on an error, they deny.
* Relationship-based authorisation engines (the Zanzibar class) pre-compute allowed objects for the hard cases.
* About 16 percent of business-critical data is overshared, by vendor estimate, reported second-hand.
* One major vendor's fix is a scoped-search allowlist. Its own documentation calls it a stopgap.
**Embeddings**
* vec2text rebuilt 92 percent of short inputs exactly from their embeddings (EMNLP 2023, reproduced 2025).
* This result underpins the embedding risk named in the OWASP list for language model applications.
**Business definitions**
* Ontology-checked querying raised accuracy from 54 to 72 percent, with refusals on about 8 percent of questions.
* A vendor's paired test: 98 to 100 percent accuracy through definitions on the modelled scope \[vendor].
* In the same test, text-to-SQL scored 84 to 90 percent \[vendor].
* The definitions covered 72.7 percent of the full question set in that test \[vendor].
* A semantic-document approach added 17 to 23 points across three frontier models \[vendor].
* On the hardest public text-to-SQL benchmark, frontier models score from single digits to the low twenties, in percent.
* On that benchmark's predecessor, the same models score above 90 percent.
**Storage**
* pgvector at 1 million vectors answers in under 20 milliseconds, with above 95 percent recall.
* pgvector index builds get painful around 2 million vectors.
* Scale-out Postgres extensions reach roughly 50 million vectors.
* One comparison shows large latency and throughput gains for scale-out Postgres at 99 percent recall \[vendor].
* At 100 million vectors with filters selecting 20 percent, Postgres p99 latency runs several times a dedicated engine's.
* Vector storage on object stores is generally available at up to 90 percent lower cost \[vendor].
* Cheap object storage favours more, smaller, purpose-scoped indexes over one large one.
**Freshness**
* One documented requirement cost about $400 a month as a five-minute batch, and $7,600 as a stream.
**Re-embedding**
* A full re-embed takes hours on multi-GPU hardware, or four figures in API fees.
* Evaluate embedding models each quarter. Migrate only for test gains above a stated threshold. 5 percent is the working figure.
* Converting embeddings between models is published at roughly 100 times cheaper than re-embedding. Test it first.
**Refusals and checks**
* Refused: a widely circulated projection about agentic analytics projects failing by 2028. No primary source was found.
* Re-verify each quarter: vector search availability on the main platforms.
* Re-verify each quarter: adoption of the semantic interchange specification.
* Re-verify each quarter: embedding model releases that would trigger a re-embed.