Agent data engineering

The agent data engineering layer (R14) is one of the fourteen layers you already own. It prepares the data that agents read and remember. Little of it exists today. Organisations have data catalogs, master data tools, document capture and a few retrieval pilots. For agents, this layer must provide data that is right, permissioned and deletable. Well-chosen data with a named owner beats a larger pile of data. Each document is read well and keeps its source. Every copy carries the original's access rules. What an agent remembers is checked before it is kept. And deleting a record reaches every copy made from it.

What sits in this layer today

This layer is mostly new for agents. The table shows what four typical estates already have, and what is missing. Vendors in brackets are examples, not a claim about any named organisation.

Standards and interfaces: CIM, and ISO 19650 for asset information, in utilities. FHIR DocumentReference, IHE XDS metadata, CDA, and retention and disposal authorities in hospitals. KKS and RDS-PP (IEC 81346), ISA-5.1 tags, ISO 8000 and ISO 15926 in energy and mining. The full estates are on the utilities, healthcare and energy pages.

Some vendors are filling the gap from their side. SAP says its utilities data products are in its Business Data Cloud from Q1 2026 [vendor]. It says meter reading and billing coverage follows by Q4 2026 (ASUG, 27 May 2026) [vendor].

What agents add

This layer turns raw records into data an agent can use safely. The source records sit on the data platform (R02). From data to memory traces the whole path. Knowledge and memory covers what an agent should remember. Freshness comes from change data captureChange data capture (CDC). Streaming each change in a database as it happens, so copies, including agent indexes, stay current.. It streams each database change as it happens. LineageLineage. The chain of steps data went through to reach its current form. runs from every answer back to its source passage.

layer · architecture

The curation pipeline with provenance end to end

Layout-aware parsing attaches provenance at page and cell level; it survives chunking, versioned embedding, permission-carrying indexing, and retrieval into every cited answer.

1. 01System

   System

   Source

   Owned corpus; authenticity at ingestion

2. 02Control

   Control

   Layout-aware parse

   Provenance at page and cell level

3. 03System

   System

   Chunk + enrich

   Per-corpus evaluated; contextual enrichment

4. 04System

   System

   Versioned embeddings

   Blue/green migration; drift monitored

5. 05Control

   Control

   Permission-carrying index

   ACLs in metadata; erasure exclusion sets

6. 06Evidence

   Evidence

   Kinds of memory

   Writes quarantined before durable promotion

7. 07Agent

   Agent

   Cited answer

   Span-level lineage to source

Derived artifacts inherit the strictest source classification; deletion cascades or it did not happen.

Diagram description: Agent data engineering pipeline from source through layout-aware parsing with provenance, contextual chunking, versioned embedding, permission-carrying index, pre-filtered retrieval, to span-cited answers, with freshness, quarantine, and erasure rails. The map contains Source: Owned corpus; authenticity at ingestion; Layout-aware parse: Provenance at page and cell level; Chunk + enrich: Per-corpus evaluated; contextual enrichment; Versioned embeddings: Blue/green migration; drift monitored; Permission-carrying index: ACLs in metadata; erasure exclusion sets; Kinds of memory: Writes quarantined before durable promotion; Cited answer: Span-level lineage to source. Its connections are source to parse; parse to chunk; chunk to embed; embed to index; index to answer for pre-filtered retrieval; memory to index for promoted writes only. Important boundary: Derived artifacts inherit the strictest source classification; deletion cascades or it did not happen.

Chosen data beats more data

A curated domain knowledge base cut hallucinationHallucination. When a model states something false with confidence. Grounding and refusal rules reduce it; verification catches the rest., confident false statements, from 35 percent to 6 percent. In another study, a collection grew from 54 to 1,128 uncurated documents. Accuracy fell from 75 percent to under 40 percent. Narrowing the collection to the domain fixed it. More data made the system worse.

So every collection an agent depends on has a named owner. CurationCuration. Preparing data for a specific purpose before agents use it, with a named owner responsible for its quality. means preparing data for a purpose, with someone answerable for its quality. Curation work is sized to risk. Automated checks, with calibrated judges and sampled audits, cover volume. People cover the cases where liability demands it.

Some content covers many aspects. An item can then be stored as several purpose-specific embeddings. Each is indexed apart and queried by the view that matches the question. Context patterns tests when this helps and when it does not. Measure with metrics that reward coverage, such as alpha-nDCG or sub-question coverage. Never use plain recall.

Read each document well and keep its source

Even the best document parsers lose at least 14 percent of retrieval performance against the true structure (OHR-Bench, ICCV 2025). Nothing later recovers what parsing dropped. So parse with the page layout in view. Attach provenanceProvenance. The record of where a piece of information came from, carried with it so every answer can be traced back to its source., the record of where each piece came from, at that moment. Check parsing quality by sampling, for as long as the collection is in use. Where data cannot leave your network, use an open-source parser that keeps provenance, under a permissive licence.

Test the split for each collection

ChunkingChunking. Splitting documents into pieces small enough for retrieval. How you split changes what the agent finds. splits documents into pieces an agent can retrieve. Gains from splitting by meaning vary across collections. Fixed-size pieces often match them. Page-level pieces did best on average in one vendor test [vendor]. So test each collection against its own real questions.

Contextual enrichment adds the surrounding context to each piece. It is the best-evidenced upgrade. It cut retrieval failures by 49 percent, and by 67 percent with reranking [vendor]. The published cost floor is about $1.02 per million document tokens, with caching.

Re-embedding discipline

An embeddingEmbedding. A numeric fingerprint of a piece of text that lets a computer find similar meaning quickly. Embeddings are derived data and inherit the protections of what they came from. is the numeric fingerprint that makes text searchable by meaning. Choose an embedding model on a public multilingual benchmark, not on vendor claims. Where storage is the limit, use techniques that shorten the vectors.

Upgrading the embedding model forces a full re-embed of the collection. For a standard collection, that takes more than ten hours on eight high-end GPUs. Through an API, it costs roughly $166 to $1,079. The discipline has five parts.

1. Stamp every vector with its model version.
2. Migrate indexes blue/green. Build the new index beside the old one, then switch.
3. Evaluate embedding models every quarter.
4. Migrate only when the evaluation gain passes a stated threshold.
5. Test cross-model conversion first. It is published as roughly 100 times cheaper than a full re-embed.

Storing vectors in object storage costs up to 90 percent less [vendor]. That favours more, smaller indexes, each scoped to one purpose.

Every copy carries the source's access rules

Access control lists go into the metadata of every piece and every vector. Retrieval filters on the caller's live identity before it searches. Embeddings can be turned back into text. Exact reconstruction of 92 percent of short inputs has been shown and reproduced. So vectors inherit their sources' classification, access rules and privacy duties.

The delegation chain decides whose rights apply. When the agent acts for a person, the lesser right wins.

One gap is still open. A summary across several documents, an extracted memory or a cluster label draws on many sources. No full published answer exists for its permissions. Current practice stamps the most restrictive mix of the source permissions. It derives the item again when an access rule changes. This is emerging practice, not a settled pattern.

Check every write to memory

Memory is an attack surface. Poisoning under 0.1 percent of a memory or knowledge base achieved over 80 percent attack success (AgentPoison, NeurIPS 2024). Memory injection through queries alone reached 98.2 percent success, with no write access (MINJA, 2025). The current vendor taxonomy of failure modes names memory poisoning.

So provenance goes on every write. A proposed memory waits in quarantine before it is promotedPromotion and demotion. Promotion moves a proven improvement into production. Demotion pulls it back out when it stops passing. to durable memory. People or evaluations approve that promotion. The quarantine rate is a standing measure.

Agents use working, episodic, semantic and procedural memory. What an agent should remember explains each kind. Writes happen during sessions, and consolidation happens between them. Treat any vendor's names for kinds of memory as a product detail.

Deleting a record reaches every copy

Deletion must reach raw logs, derived memories and vectors. The published mechanics have three parts. Deleted vectors go on an exclusion list, checked before every similarity search. Purpose-scoped namespaces limit how far a deletion must reach. And deletion is an audited operation that systems call, not a manual runbook.

Copies inherit residency. Embeddings, memories and summaries built from in-region data are in-region data. Bitemporal facts record when an event happened and when it was recorded. A replaced fact can then be retired without losing the history behind an earlier decision.

What must be true before an agent touches it

Before an agent reads or writes this data, these hold. Each check sits in the data path or the tool gateway, not in the prompt. Enforcement outside the model explains the points.

• Every collection has a named owner. The owner holds a test set of real questions with known answers.
• Retrieval filters on live identity. The filter runs before the search. The agent never sees what the caller may not see.
• Provenance runs from parse to answer. Every answer can show the page and passage it came from.
• Memory writes are quarantined. Promotion to durable memory needs a person or an evaluation gate.
• Deletion is tested. An erasure request reaches every copy and returns receipts before go-live.
• Classification and residency travel with every copy. Governance, risk and sovereignty (R11) sets the classes.
• Changes re-run the checks. A new data source or a new memory policy is a change trigger on the agent lifecycle. It re-runs the evaluations and the risk profile.
• You run your own acceptance suite. Cover local facts, semantic recall, exact identifiers and multi-hop questions. Add facts that change over time, whole-collection synthesis, unanswerable questions, deletions and access rules. Measure correctness, evidence recall, faithfulness and 95th-percentile latency. Also measure cost, ingestion lag and permission leakage.

Choices and their verdicts

Each choice below is the common choice, tested against the simplest credible alternative. The full records are in the decision catalog.

• One universal index, or indexes scoped to each use case (decision record CD-11). Both. A permission-aware universal index serves discovery and questions across silos. Curated, purpose-scoped indexes serve high-stakes tasks.
• A graph database, or embedding retrieval (decision record CD-1). A routed hybrid, vector-first. Add a graph only for fixed, repeatable lookups. Every generated edge then carries its source passage, the extraction model and version, a confidence value and a validity period.
• Purpose-specific embedding views, or matched chunking (decision record CD-25). Chunks first. Use views only on a measured win against a control with the same embedding budget. Never shape the views around the objective.
• Who curates. Every collection has a named accountable owner. A dedicated curator per domain is not supported as published fact. Accountability cannot be automated.

Concerns this layer must honour

This layer shares ownership of one of the ten cross-cutting concerns with the data platform. It carries duties for six more.

• Grounding (owned here, with the data platform). Curated collections with owners, contextual enrichment and fresh data.
• Traceability and audit. Provenance runs from parse time to the claim. Memory writes are audited.
• Privacy. Embeddings count as recoverable personal data. Deletion reaches logs, memories, vectors and exclusion lists.
• Sovereignty and residency. Copies inherit the residency of their source.
• Identity and access. Access rules travel into pieces, vectors and derived items.
• Safety and oversight. Memory writes are quarantined, and promotion is gated.
• Resilience. Everything can be rebuilt from the raw source. A full re-index is the recovery baseline.

Evidence and limits

The parsing loss, the poisoning rates, the embedding reconstruction and the curation results are peer-reviewed. The enrichment gains, the chunking comparisons and the storage costs are vendor-published and marked.

Every memory-service benchmark in circulation is written by a vendor and disputed by other vendors. This page cites none of them. Run your own acceptance suite instead.

Permissions for items drawn from many sources remain an open problem. Current practice is emerging, not settled.

Next step

See this layer at work in investigating recurring audit issues. This is the last of the fourteen layers. To start building, go to the intelligence ladder.

Explore this topic

• Agent data engineering findings
• Sources
• Products named for orientation, on the one-page wall chart

Sources named on this page. OHR-Bench, ICCV 2025, the International Conference on Computer Vision. AgentPoison, NeurIPS 2024, the Conference on Neural Information Processing Systems. MINJA, 2025. The enrichment, page-level chunking and object-storage figures are vendor-published [vendor]. The metric alpha-nDCG is a coverage-aware form of normalised discounted cumulative gain.

Refusals. Every memory-service benchmark now in circulation is vendor-authored and publicly disputed between vendors. This guide cites none of them.

Re-verify. Whether any complete solution is published for the permissions of items derived from many sources. Whether independent memory benchmarks emerge.