Agent data engineering

Target state

In short: Agents answer only as well as their data, so every data set gets a named owner, a traceable source, and a working delete.

A curationCuration. Preparing data for a specific purpose before agents use it, with a named owner responsible for its quality. pipeline (data prepared for a purpose, with someone responsible for its quality) with named ownership per corpus, meaning each collection of documents. Layout-aware parsing attaches provenanceProvenance. The record of where a piece of information came from, carried with it so every answer can be traced back to its source., the record of where each piece came from, at parse time. ChunkingChunking. Splitting documents into pieces small enough for retrieval. How you split changes what the agent finds., the splitting of documents into retrievable pieces, is evaluated per corpus rather than adopted as a universal recipe. Contextual enrichment is added where retrievalRetrieval. Finding the few relevant pieces of company information for a task and placing them in front of the model. quality matters. EmbeddingsEmbedding. A numeric fingerprint of a piece of text that lets a computer find similar meaning quickly. Embeddings are derived data and inherit the protections of what they came from. (the numeric fingerprints that make text searchable by meaning) are versioned and monitored for drift. IndexesVector index. A database of embeddings built for finding similar meaning fast. The storage behind most retrieval. carry permissions. Memory tiersAgent memory. What an agent keeps between tasks, from the current conversation up to long-term knowledge about customers or assets. The longer it is kept, the more privacy and retention duties it carries. have write governance, time-to-live, and erasureErasure cascade. When a record is deleted, every copy derived from it (embeddings, memories, traces, test data) is deleted too.. Freshness is fed by change data captureChange data capture (CDC). Streaming each change in a database as it happens, so copies, including agent indexes, stay current., which streams each database change as it happens. Span-level lineageLineage. The chain of steps data went through to reach its current form. runs from every answer to its source. Curation labour is sized to risk. It is automated, with calibrated judgesJudge (model judge). An AI model used to grade another model's answers at scale. Checked regularly against human graders, and never allowed to grade its own work. and sampled audits, where volume demands it. It is human where liability demands it. It always sits under a named accountable owner.

layer · architecture

The curation pipeline with provenance end to end

Layout-aware parsing attaches provenance at page and cell level; it survives chunking, versioned embedding, permission-carrying indexing, and retrieval into every cited answer.

1. 01System

   System

   Source

   Owned corpus; authenticity at ingestion

2. 02Control

   Control

   Layout-aware parse

   Provenance at page and cell level

3. 03System

   System

   Chunk + enrich

   Per-corpus evaluated; contextual enrichment

4. 04System

   System

   Versioned embeddings

   Blue/green migration; drift monitored

5. 05Control

   Control

   Permission-carrying index

   ACLs in metadata; erasure exclusion sets

6. 06Evidence

   Evidence

   Memory tiers

   Writes quarantined before durable promotion

7. 07Agent

   Agent

   Cited answer

   Span-level lineage to source

Derived artifacts inherit the strictest source classification; deletion cascades or it did not happen.

Diagram description: Agent data engineering pipeline from source through layout-aware parsing with provenance, contextual chunking, versioned embedding, permission-carrying index, pre-filtered retrieval, to span-cited answers, with freshness, quarantine, and erasure rails. The map contains Source: Owned corpus; authenticity at ingestion; Layout-aware parse: Provenance at page and cell level; Chunk + enrich: Per-corpus evaluated; contextual enrichment; Versioned embeddings: Blue/green migration; drift monitored; Permission-carrying index: ACLs in metadata; erasure exclusion sets; Memory tiers: Writes quarantined before durable promotion; Cited answer: Span-level lineage to source. Its connections are source to parse; parse to chunk; chunk to embed; embed to index; index to answer for pre-filtered retrieval; memory to index for promoted writes only. Important boundary: Derived artifacts inherit the strictest source classification; deletion cascades or it did not happen.

Reviewed 2026-08-20·Sources:r14-agent-data-engineering findings

Mechanisms

Parsing caps everything downstream

In short: If a document is read badly at the start, nothing later can recover what was lost.

Even the best document parsers lose at least 14 percent of retrieval performance compared with ground-truth structure (OHR-Bench, ICCV 2025, the International Conference on Computer Vision). Nothing downstream recovers what parsing dropped. That makes layout-aware parsing with provenance the floor rather than an optimisation. It also makes parsing fidelity a permanent sampled metric rather than a one-time acceptance test. Where data cannot leave the perimeter, the neutral default is a provenance-preserving open-source parser under a permissive licence.

Chunking is evaluated, never inherited

In short: There is no universal best way to split documents; test each collection against its own real questions.

Gains from semantic chunking are inconsistent across corpora. Fixed-size chunking frequently matches it. Page-level chunking was best on average in one vendor benchmarkBenchmark. A public test used to compare models or agents. Useful for orientation; scores can be gamed, so they are claims rather than measurements. [vendor]. The rule that survives: evaluate per corpus, against that corpus's real questions. The best-evidenced upgrade is contextual enrichment. It reduced retrieval failures by 49 percent, and by 67 percent when combined with reranking [vendor]. The published cost floor is about $1.02 per million document tokensToken. The unit AI usage is measured and billed in, roughly three-quarters of a word. with caching, where a token is the unit AI usage is billed in.

Embedding operations and the re-embed budget

In short: Changing how text is turned into searchable numbers is costly, so version everything and switch only for proven gains.

Select an embedding model on a public multilingual benchmark rather than on vendor claims. Use dimension-truncation techniques where storage is the constraint. Upgrading the embedding model forces a full re-embed of the corpus. Reference costs run from more than ten hours on eight high-end graphics processing units (GPUs) to roughly $166 to $1,079 through an application programming interface (API)API. A defined way for software to talk to other software. Agents reach systems through governed APIs rather than through screens. for a standard corpus. The discipline has five parts. Version-stamp every vector. Run blue/green index migrations. Evaluate embedding models quarterly. Migrate only for evaluation (eval)Eval (evaluation suite). A set of test tasks with known right answers, written by people who know the work, used to check an agent before and after any change. gains above a stated threshold, meaning gains on a test set with known right answers. Test cross-model conversion, published at roughly 100 times cheaper, before committing to a full re-embed. Object-store vector storage at up to 90 percent lower cost [vendor] favours more, smaller, purpose-scoped indexes.

Memory is an attack surface with quantified exposure

In short: Very little poisoned data can hijack an agent's memory, so each write is checked before it sticks.

Poisoning under 0.1 percent of a memory or knowledge base achieved over 80 percent attack success (AgentPoison, NeurIPS 2024, the Conference on Neural Information Processing Systems). Query-only memory injectionPrompt injection. An attack where instructions hidden in content the agent reads (an email, a web page, a document) trick it into doing something its user never asked for. reached 98.2 percent success without any write access (MINJA, 2025). Memory poisoning is named as a first-class failure mode in the current vendor failure-mode taxonomy. The controls follow directly. Provenance goes on every write. Memories are quarantined before promotionPromotion and demotion. Promotion moves a proven improvement into production. Demotion pulls it back out when it stops passing. to durable tiers, with human or eval gates on that promotion. Poisoning is monitored, and memory-write quarantine rates are a standing metric. The functional tiers are working, episodic, semantic, and procedural. Hot-path writes happen during sessions and background consolidation happens between them. Treat any vendor's tier naming as a product detail rather than an architecture.

Permissions must travel with derivations

In short: Anything made from a document inherits that document's access rules, because the copies can be turned back into the original.

Access control lists (ACLs)ACL-aware retrieval. Retrieval that checks the requester's permissions before returning information, so an agent cannot surface documents its user could not open. go into chunk and vector metadata, and retrieval pre-filters on the caller's live identityAgent identity. A registered account for an agent with a named owner, limited permissions, and credentials that expire, so every action can be traced to who asked.. This is non-negotiable because embeddings are recoverable text: 92 percent exact reconstruction of short inputs has been demonstrated and reproduced. So vectors inherit their sources' classificationClassification (data). A label for how sensitive data is. Anything derived from data inherits the strictest label of its sources. (sensitivity label), ACLs, and privacy obligations. The open gap, stated rather than solved: many-to-one derived artifacts (cross-document summaries, extracted memories, cluster labels) have no fully published permissions solution. Current practice stamps the intersection, meaning the most restrictive combination of the source permissions, and re-derives when an ACL changes. That is emerging practitioner-grade work, not a settled pattern.

Erasure cascades or it did not happen

In short: Deleting a record must also delete every copy made from it, or the deletion did not really happen.

Deletion must reach raw logs, derived memories, and vectors. The published mechanics have three parts. Deleted-vector exclusion sets are consulted before approximate nearest neighbour (ANN) queries, the fast similarity search behind retrieval. Purpose-scoped namespaces give deletion a bounded blast radius. Deletion is exposed as an audited callable operation rather than a manual runbook. Derived artifacts inherit residencyData residency and sovereignty. Rules about which country or region data may be stored and processed in, including the copies created for agents.: embeddings, memories, and summaries built from in-region data are in-region data. Bitemporal facts separate event time from ingestion time. They let a superseded fact be invalidated without destroying the history that explains an earlier decision.

Curation is the accuracy lever, and it is measurable

In short: Well-chosen data beats more data, and the gain shows up in how often the agent is wrong.

A curated domain knowledge base cut hallucinationHallucination. When a model states something false with confidence. Grounding and refusal rules reduce it; verification catches the rest. (confident false statements) from 35 percent to 6 percent. Growing a corpus from 54 to 1,128 uncurated documents dropped accuracy from 75 percent to under 40 percent until domain scoping fixed it: more data made the system worse. The trade is formalised as coverage versus trust. Where content is aspect-rich, an item can be represented through several purpose-specific embeddings, each indexed separately and queried by the view that matches the question. That improves precision per question type, at embedding cost rather than modelModel. The AI engine that reads and writes text. On its own it has no access to your systems; everything built around it decides what it may do. cost. The taxonomy of views is designed with a model from the corpus and hardened by a human; classical machinery executes at scale, and a model names the discovered clusters. The guide tested this pattern in five rounds on public corpora (the research pages): several vectors per document beat one decisively, purpose views beat matched chunks only where queries target one aspect and lose where they concern whole documents, and conditioning the views on the objective made retrieval worse in every form tested. The design that survived is the recommended approach. Measure with coverage-aware metrics such as alpha-nDCG (normalised discounted cumulative gain, coverage-aware form) or sub-question coverage, never plain recall.

Design decisions

• One universal index vs use-case-scoped (CD-11), a challenged defaultChallenged default. A technology choice most people treat as obvious, re-examined here against the simplest credible alternative on evidence and cost. about whether to build one big search index or several. The answer is two layers. A permission-aware universal index serves discovery and cross-silo questions. Purpose-scoped curated indexes serve high-stakes tasks.
• Graph database vs embedding retrieval (CD-1), a challenged default about whether a knowledge graph should replace vector search. The answer is a routed hybrid, vector-first, with a selective provenance-linked graph only for deterministicDeterministic. Always gives the same answer for the same input by following fixed rules. The opposite of probabilistic, where the answer can vary. traversal (fixed-rule, repeatable lookups). If you add a graph, every generated edge carries a source-span identifier (ID), the extraction model and its version, a confidence value, and a validity interval.
• Purpose-specific embedding views vs matched chunking (CD-25), a challenged default about whether to split documents by purpose or by fixed windows. The answer is chunks first, views only on a measured win against a control with the same embedding budget, and never a view taxonomy conditioned on the objective.
• Corpus ownership: every corpus an agent depends on has a named accountable owner. Curation labour is sized to risk. The dedicated-curator-per-domain form is not supported as published fact, and accountability cannot be automated.

Cross-cutting concerns

Evidence and limits

The parsing loss, the poisoning success rates, the embedding reconstruction, and the corpus-curation accuracy results are peer-reviewed. The contextual-enrichment improvements, the chunking comparisons, and the storage economics are vendor-published and flagged. One standing refusal: every memory-service benchmark now in circulation is vendor-authored and publicly disputed between vendors, so this guide cites none of them. Run your own acceptance suite instead. Build that suite to cover local facts, semantic recall, exact identifiers, multi-hop questions, temporal contradictions, corpus-global synthesis, unanswerable questions, deletions, and ACL enforcement. Measure correctness, evidence recall, faithfulness, 95th-percentile (p95) latencyLatency. How long a request takes to come back., cost, ingestion lag, and permission leakage. Re-verify two things: whether any complete solution to many-to-one derived-artifact permissions is published, and whether independent memory benchmarks emerge.

The research behind this page

• Agent data engineering findings
• Research brief
• Vendor landscape
• Sources