The data-to-memory pipeline

How company data becomes a cited answer and an agent memory in nine stages, and the rule that every copy keeps the original's access, privacy and deletion duties.

An agent answers a question from company documents. Can you show where each part of the answer came from? And will a deleted record stay deleted in everything built from it? Company data reaches a cited answer in nine stages, in three groups. First, get the data ready: parse, chunk, embed and index it. Then answer from it: retrieve with the right to refuse, and cite. Afterwards, write memory through quarantine, refresh when sources change, and erase in cascade. Every copy made on the way keeps the original's access, privacy and deletion duties.

The deep detail sits in two of the fourteen layers: the data platform (R02) and agent data engineering (R14).

Get the data ready

1. Parse with provenance. Parsing reads each document and keeps its source trail. Provenance is a record of exactly where each piece came from, down to the page and table cell. Even the best parsers lose some search quality against the true document structure. So check parsing quality on a sample, for as long as the pipeline runs.

2. Chunk and add context. Documents are split into chunks small enough to find and use. No single way of splitting works for every collection, so test each one. Adding a short note of context to each chunk is the best-tested improvement [vendor].

3. Embed with versions. An embedding is a list of numbers that captures meaning, used to find similar text. Each embedding records which model version made it. An upgrade runs the new version beside the old one. Switch only when a stated eval gain is met. Re-embedding everything takes hours and real money.

4. Index with permissions inside. Each source's access list is copied into the search index. Every search is filtered on the caller's live identity before it runs. If the permission sync fails, the search returns nothing rather than too much. Filtering after the search is a mistake. It leaks counts and starves the results.

Answer from it

5. Retrieve with the right to refuse. Some numbers are too important to guess. An agreed business definition of each measure, called a semantic contract, decides how they are answered. Either a fixed calculation gives the number, or the agent refuses. A silent wrong answer becomes a visible refusal.

6. Cite by construction. Each answer carries links back to the exact passages it used. This is also the most valuable part of the audit trail. The governance, risk and sovereignty layer (R11) sets out how it is kept.

Afterwards

7. Write memory through quarantine. Anything the agent writes to long-term memory carries its source. It waits in quarantine until a rule or a person promotes it. The reason is measured. Poisoning a tiny share of a memory store has taken over most of an agent's answers in tests.

8. Refresh when sources change. Change data capture streams each database change as it happens. It keeps indexes fresh to within minutes, at close to batch cost. Full real-time streaming must be justified by how often decisions are made. In one documented case, the two approaches cost $400 and $7,600 a month for the same need.

9. Erase in cascade. Deleting a record deletes every copy made from it. That means raw stores, embeddings, memories, traces and summaries. Search checks a list of deleted items first, so nothing deleted can come back. Each deletion is itself logged.

The rule behind all nine

Every copy carries the original's duties. That covers its access rules and its privacy duties. It covers the country or region it must stay in, its sensitivity label and its deletion duty. Embeddings count as data too. Short texts have been rebuilt from their embeddings 92 percent of the time. A copy built from several sources takes the strictest label among them. These duties are the sovereignty and privacy concerns on the concern matrix.

One gap is still open. A summary built from many documents has no complete permissions answer yet. Current practice gives it only the permissions that all its sources share. It is rebuilt whenever a source's access list changes. This practice is emerging, not settled.

Working, thread and long-term memory

Working memory is what the model has in front of it for the current step. It is engineering. Thread memory keeps one case or conversation going across a pause. Long-term memory keeps what is worth reusing in later tasks.

Once information outlives the task, it is a record. Ownership, consent, retention and deletion duties attach to it. So moving anything into long-term memory is a gated step, never a slow pile-up. What is worth keeping is covered on knowledge and memory.

A worked example: overdue inspections at a utility

A field supervisor asks an agent which inspections are overdue on one circuit. The agent searches asset records, inspection reports and work orders. Each report was parsed with its source page, split, embedded, and indexed with its access list. So the agent sees only what that supervisor may see.

Its answer names each report and the passage it used. The overdue count goes into a report to the regulator, so the semantic contract sends it to a fixed calculation. When a crew closes a work order, change data capture refreshes the index within minutes. More on the systems involved is on utilities.

Evidence and limits

The nine stages and their failure modes come from measured studies and practitioner reports. The result for adding context to chunks is a vendor figure. The results on memory poisoning and rebuilding text from embeddings come from research papers, under test conditions.

The cost comparison for change data capture is one documented case. The permissions answer for summaries built from many sources is emerging practice, not a standard.

Next step

See how an agent's behaviour improves without losing control on the learning flywheel.

Explore this topic

The memory-pipeline architecture
Agent data engineering findings
Data platform findings
Context patterns: the measured experiments on how to represent documents and choose what reaches the agent. They inform the chunk, embed and retrieve stages.

Figures behind the stages. Even the best parsers lose at least 14 percent of retrieval performance against the true document structure. Adding context to each chunk cut retrieval failures by 49 percent, and by 67 percent with reranking [vendor]. Reranking means sorting the first results again with a second, slower model. Poisoning less than 0.1 percent of a memory store has achieved more than 80 percent attack success. Injection through queries alone has reached 98.2 percent. Exact reconstruction of short inputs from their embeddings has been shown at a 92 percent rate. Search consults a list of deleted embeddings before any approximate nearest-neighbour query, the fast similarity search.