# Agent data engineering & memory pipelines

Author: Murali Sid (https://linkedin.com/in/muralisid)
Source: https://www.agenticarchitectureskills.com/library/layers/r14-agent-data-engineering
Published: 2026-09-18
Updated: 2026-09-18
Licence: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/
Cite as: Murali Sid. Agent data engineering & memory pipelines. Agentic Architecture Skills. https://www.agenticarchitectureskills.com/library/layers/r14-agent-data-engineering

---


Status: complete (published 2026-08-19). Scope finalized at Batch A kickoff with maintainer POV; see findings.md and sources.md.

## Scope inventory (v0) [#scope-inventory-v0]

* **Ingestion and preparation**: connectors, parsing and OCR, chunking strategy, metadata enrichment (entities, ownership, freshness), PII handling and consent at ingestion
* **Purpose-driven curation**: use-case-scoped corpora, multi-view/multi-card embedding design, LLM-guided taxonomy with classical topic modelling on embeddings, SME-in-the-loop curation workflows, knowledge hub editorial process
* **Memory feeding**: promotion policies into entity and cross-domain memory tiers, TTL and refresh, ACL propagation into indexes, index lifecycle and re-embedding operations
* **Economics and quality**: retrieval quality evaluation per view (recall and precision), embedding-vs-LLM cost engineering, data contracts with source owners

## Challenged-default candidates [#challenged-default-candidates]

Full-corpus LLM processing vs LLM-guided curation with deterministic machinery (the founding technique of this track); one universal index vs use-case-scoped multi-view indexes. Each track proposes its final list at kickoff.

## Files [#files]

`findings.md` and `sources.md` carry this track's research. The scoping brief and the vendor map were archived on 2026-08-23 (see `archive/`, decision D038).