The data-to-memory pipeline

The pipeline

In short: Data passes through nine steps on its way to an agent's answer, and every step keeps a record of where the information came from.

foundation · flow

From source data to governed memory

Permission, provenance, and classification travel with content from ingestion onward.

1. 1

   Source

   Acquire governed content.

2. 2

   Permissions

   Capture ACLs and purpose.

3. 3

   Parse

   Attach provenance immediately.

4. 4

   Classify

   Apply strictest inherited class.

5. 5

   Curate

   Select for the use case.

6. 6

   Index

   Build permission-aware retrieval.

7. 7

   Retrieve

   Filter before model access.

8. 8

   Promote

   Gate durable memory changes.

9. 9

   Erase

   Cascade deletion to derivatives.

Diagram description: Nine-stage data-to-memory pipeline covering source intake, permission capture, parsing, classification, curation, indexing, retrieval, promotion, and erasure propagation. The sequence contains 9 stages: 1, Source: Acquire governed content.; 2, Permissions: Capture ACLs and purpose.; 3, Parse: Attach provenance immediately.; 4, Classify: Apply strictest inherited class.; 5, Curate: Select for the use case.; 6, Index: Build permission-aware retrieval.; 7, Retrieve: Filter before model access.; 8, Promote: Gate durable memory changes.; 9, Erase: Cascade deletion to derivatives..

Reviewed 2026-08-20·Sources:Memory pipeline architecture

There are nine stages, each with a measured failure mode. The deep treatment is on R02, the data platform layer, and R14, the agent data engineering layer. The architectural spine:

1. Parse with provenance. Parsing that understands page layout attaches provenanceProvenance. The record of where a piece of information came from, carried with it so every answer can be traced back to its source., a record of exactly where each piece came from, at page and cell level as it reads. Even the best parsers lose at least 14 percent of retrievalRetrieval. Finding the few relevant pieces of company information for a task and placing them in front of the model. performance compared with the true document structure, so parsing fidelity is spot-checked forever.
2. Chunk per corpus, enrich contextually. Documents are split into chunksChunking. Splitting documents into pieces small enough for retrieval. How you split changes what the agent finds. small enough to retrieve, and no single splitting recipe survives evaluation across every collection. Enriching each chunk with context is the best-evidenced upgrade: 49 percent fewer retrieval failures, and 67 percent fewer with reranking [vendor].
3. Embed versioned. Every embeddingEmbedding. A numeric fingerprint of a piece of text that lets a computer find similar meaning quickly. Embeddings are derived data and inherit the protections of what they came from. (a numeric fingerprint used to find similar meaning) carries the version of the model that produced it. Upgrades run blue/green, with the new version alongside the old, and migrate only when a stated evalEval (evaluation suite). A set of test tasks with known right answers, written by people who know the work, used to check an agent before and after any change. gain is met. The reason for caution is cost: re-embedding everything takes hours on serious hardware, or a four-figure sum through a paid application programming interface (API)API. A defined way for software to talk to other software. Agents reach systems through governed APIs rather than through screens..
4. Index with permissions inside. The access control lists (ACLsACL-aware retrieval. Retrieval that checks the requester's permissions before returning information, so an agent cannot surface documents its user could not open.) of each source are carried into the vector indexVector index. A database of embeddings built for finding similar meaning fast. The storage behind most retrieval. as metadata. Retrieval is filtered on the caller's live identity before the search runs, and a failed permissions sync fails closedDegraded mode. What a system does when a part fails. For agents, that means falling back to a human queue rather than guessing., returning nothing rather than too much. Filtering after the search is the anti-pattern: it leaks counts and starves results.
5. Retrieve with refusal available. A semantic contractSemantic layer. A shared set of business definitions (what counts as revenue, or an active customer) that keeps agents and reports using the same meaning., an agreed business definition of each measure, routes high-stakes numbers either to a deterministicDeterministic. Always gives the same answer for the same input by following fixed rules. The opposite of probabilistic, where the answer can vary. answer computed by fixed rules or to a refusal. That turns silent wrong answers into visible ones.
6. Cite by construction. Answers carry lineageLineage. The chain of steps data went through to reach its current form. back to their sources at the level of individual text spans. This is also the single most valuable component of the evidence posture, as R11, the governance, risk, and sovereignty layer, sets out.
7. Write memory through quarantine. Anything written to durable agent memoryAgent memory. What an agent keeps between tasks, from the current conversation up to long-term knowledge about customers or assets. The longer it is kept, the more privacy and retention duties it carries. carries provenance and waits in quarantine until it is promotedPromotion and demotion. Promotion moves a proven improvement into production. Demotion pulls it back out when it stops passing.. The reason is measured. Poisoning less than 0.1 percent of a memory store has achieved more than 80 percent attack success. InjectionPrompt injection. An attack where instructions hidden in content the agent reads (an email, a web page, a document) trick it into doing something its user never asked for. through queries alone has reached 98.2 percent.
8. Refresh by CDC. Change data capture (CDC)Change data capture (CDC). Streaming each change in a database as it happens, so copies, including agent indexes, stay current., which streams each database change as it happens, keeps indexes fresh to within minutes at close to batch cost. Full streaming has to be earned by how often decisions are made, not chosen by default. In one documented case the two approaches cost $400 versus $7,600 per month for the same stated requirement.
9. Erase in cascade. Deleting a record triggers an erasure cascadeErasure cascade. When a record is deleted, every copy derived from it (embeddings, memories, traces, test data) is deleted too.: the deletion reaches raw stores, embeddings, memories, tracesTrace. The step-by-step log of one agent run: what it read, what it called, and what it decided., and every derived artifact. Search consults a list of deleted embeddings before any approximate-nearest-neighbour query (the fast similarity search), so deleted items cannot resurface. Deletion itself is exposed as an audited operation.

The two rules that make it governance, not plumbing

In short: Searchable copies of your data are still your data, and multi-document summaries have no complete permissions answer yet.

Embeddings are recoverable data. Exact reconstruction of short inputs from their embeddings has been demonstrated at a 92 percent rate. So embeddings inherit everything from their sources, in full. That means the sensitivity classificationClassification (data). A label for how sensitive data is. Anything derived from data inherits the strictest label of its sources., the access control lists, the data residencyData residency and sovereignty. Rules about which country or region data may be stored and processed in, including the copies created for agents. rules (which country or region the data may sit in), and the erasure obligations. Any derived artifact inherits the strictest classification among its sources.

The open gap, stated rather than papered over: artifacts built from many sources at once (cross-document summaries, extracted memories) have no complete published permissions solution. Current practice gives such an artifact only the permissions that all of its sources share (the intersection), and rebuilds it whenever a source's access list changes. This is emerging, practitioner-grade practice, and it is flagged as such.

Memory tiers

In short: What an agent holds during one task is an engineering matter; what it keeps afterwards is a record with ownership and privacy duties attached.

Working contextContext window. The model's working memory for one task. It is limited in size, so what goes in and what gets left out is a design decision., the model's working memory for one task, is engineering. Durable memory is a record. The line sits where information persists past the end of a session. From that point, ownership, consent, retention, and erasure obligations attach. Promotion into a durable tier is a gated act, never an accumulation.

foundation · stack

Memory persistence increases obligation

More durable memory is not more intelligence; it is more ownership, consent, retention, and erasure work.

Increasing persistence and obligation↓

1. 1

   M1 Thread

   The current conversation or run.

2. 2

   M2 Retrieved knowledge

   Permission-aware evidence for the current turn.

   Control: Provenance on every span

3. 3

   M3 Session

   Bounded continuity across hours or days.

   Control: Residency and retention

4. 4

   M4 Entity memory

   Durable profiles of customers, assets, cases, or employees.

   Control: Consent, ownership, and erasure

5. 5

   M5 Cross-domain

   Shared memory with the broadest blast radius.

   Control: Strictest governance

Diagram description: Five-tier memory stack from thread and retrieved knowledge through session, entity, and cross-domain memory, with obligations increasing as persistence rises. The sequence contains 5 stages: 1, M1 Thread: The current conversation or run.; 2, M2 Retrieved knowledge: Permission-aware evidence for the current turn., followed by the Provenance on every span gate; 3, M3 Session: Bounded continuity across hours or days., followed by the Residency and retention gate; 4, M4 Entity memory: Durable profiles of customers, assets, cases, or employees., followed by the Consent, ownership, and erasure gate; 5, M5 Cross-domain: Shared memory with the broadest blast radius., followed by the Strictest governance gate.

Reviewed 2026-08-20·Sources:Memory pipeline architecture

The research behind this page

• The memory-pipeline architecture
• Agent data engineering findings
• Data platform findings
• The multi-card retrieval experiments, the measured evidence behind the purpose-scoped curation step