The papers behind the patterns

The published work behind each context pattern, grouped by the question it answers. Each entry says what the paper shows, and whether our results agree, disagree or set a limit.

The context patterns rest on our own tests and on published work. We asked which papers support each pattern, which contradict it, and which set its limits. This page lists that work by the question it answers. Each row says what the paper showed in plain words. It then says how our results relate. No pattern rests on our experiments alone. Each has independent published support. Vendor results are marked [vendor].

For how these ideas fit an agent, see step 2 of the intelligence ladder: context and knowledge and memory. The engineering is on the data-to-memory pipeline and the agent data engineering layer (R14). The patterns themselves are on context patterns for agents.

Several vectors per document, and the size of each piece

Paper	What it shows	How our results relate
Weller, Boratko, Naim, Lee. On the Theoretical Limitations of Embedding-Based Retrieval. arXiv:2508.21038, Aug 2025. paper	For a vector of fixed size, some sets of top results can never be returned by a one-vector index. Leading embedding models fail the LIMIT test set built to show it.	We agree. One pooled vector scored 0.314 on LIMIT-small. Per-attribute cards scored 0.988. Fine blind windows scored 0.924.
Khattab and Zaharia. ColBERT. SIGIR 2020. paper. Santhanam et al. ColBERTv2. NAACL 2022. paper	Search with one vector per word, called late interaction. The standing criticism is storage, at 10 to 30 times that of one-vector search.	This sets a limit on our design. Purpose cards sit between one vector and one per word. Storage depends on the documents: 44 cards per document on LIMIT, 2.8 on SciFact.
Zhang et al. Multi-View Document Representation Learning. ACL 2022. paper	Several learned vectors per document, one for each kind of query.	The closest academic name for the idea. There the views are learned and fixed, not shaped by a goal.
Kong et al. Multi-Aspect Dense Retrieval. KDD 2022.	Separate vectors for each topic, taken from product data fields.	Same family. The topics are fixed, set by the data fields available.
Dong et al. Multi-view Content-aware Indexing. EMNLP Findings 2024. paper	Each section is indexed three fixed ways: raw text, keywords and a summary.	A fixed multi-view index. It fits our finding that card types built from the documents are the lasting positive.
Chen et al. Dense X Retrieval. EMNLP 2024. paper	Indexing by single statements, called propositions. Recall at 20 rose by +10.1 in the unsupervised setting.	Piece size matters and depends on the workload. That is what our SciFact result shows.
Duarte et al. LumberChunker. EMNLP Findings 2024. paper. Zhong et al. Mix-of-Granularity. 2024. paper	A model places the chunk boundaries. Or a router picks among chunk sizes.	Both pick one piece size for everything, and both need a model to read every item. Our approach keeps the model off the per-item path.
Anthropic. Contextual Retrieval. Sep 2024 [vendor]. post	Adds a short situating context to each chunk before embedding. It costs about $1.02 per million document tokens with caching [vendor].	The published floor for a model touching every chunk once. The two-pass design avoids that cost by design.
Weller et al. When do Generative Query and Document Expansions Fail? EACL 2024. paper	Expanding queries or documents with model-written text helps weak retrievers. It can hurt strong ones.	It fits our finding that purpose cards hurt on SciFact next to a strong hybrid.

Paper: Weller, Boratko, Naim, Lee. On the Theoretical Limitations of Embedding-Based Retrieval. arXiv:2508.21038, Aug 2025. paper
What it shows: For a vector of fixed size, some sets of top results can never be returned by a one-vector index. Leading embedding models fail the LIMIT test set built to show it.
How our results relate: We agree. One pooled vector scored 0.314 on LIMIT-small. Per-attribute cards scored 0.988. Fine blind windows scored 0.924.

Paper: Khattab and Zaharia. ColBERT. SIGIR 2020. paper. Santhanam et al. ColBERTv2. NAACL 2022. paper
What it shows: Search with one vector per word, called late interaction. The standing criticism is storage, at 10 to 30 times that of one-vector search.
How our results relate: This sets a limit on our design. Purpose cards sit between one vector and one per word. Storage depends on the documents: 44 cards per document on LIMIT, 2.8 on SciFact.

Paper: Zhang et al. Multi-View Document Representation Learning. ACL 2022. paper
What it shows: Several learned vectors per document, one for each kind of query.
How our results relate: The closest academic name for the idea. There the views are learned and fixed, not shaped by a goal.

Paper: Kong et al. Multi-Aspect Dense Retrieval. KDD 2022.
What it shows: Separate vectors for each topic, taken from product data fields.
How our results relate: Same family. The topics are fixed, set by the data fields available.

Paper: Dong et al. Multi-view Content-aware Indexing. EMNLP Findings 2024. paper
What it shows: Each section is indexed three fixed ways: raw text, keywords and a summary.
How our results relate: A fixed multi-view index. It fits our finding that card types built from the documents are the lasting positive.

Paper: Chen et al. Dense X Retrieval. EMNLP 2024. paper
What it shows: Indexing by single statements, called propositions. Recall at 20 rose by +10.1 in the unsupervised setting.
How our results relate: Piece size matters and depends on the workload. That is what our SciFact result shows.

Paper: Duarte et al. LumberChunker. EMNLP Findings 2024. paper. Zhong et al. Mix-of-Granularity. 2024. paper
What it shows: A model places the chunk boundaries. Or a router picks among chunk sizes.
How our results relate: Both pick one piece size for everything, and both need a model to read every item. Our approach keeps the model off the per-item path.

Paper: Anthropic. Contextual Retrieval. Sep 2024 [vendor]. post
What it shows: Adds a short situating context to each chunk before embedding. It costs about $1.02 per million document tokens with caching [vendor].
How our results relate: The published floor for a model touching every chunk once. The two-pass design avoids that cost by design.

Paper: Weller et al. When do Generative Query and Document Expansions Fail? EACL 2024. paper
What it shows: Expanding queries or documents with model-written text helps weak retrievers. It can hurt strong ones.
How our results relate: It fits our finding that purpose cards hurt on SciFact next to a strong hybrid.

Using an instruction or goal to shape search

Paper	What it shows	How our results relate
Su et al. INSTRUCTOR. ACL Findings 2023. paper	One encoder that makes task-specific vectors from an instruction.	It does at question time what goal-shaped cards tried to do at build time.
Weller et al. FollowIR. EMNLP 2024. paper	A test set with two sets of human judgements per query, before and after the instruction changes. It also defines the p-MRR score.	Our real-data test in round 5. Re-picking cards per instruction scored p-MRR about zero.
Weller et al. Promptriever. ICLR 2025. paper	Retrievers trained to follow instructions, as language models do.	The strong baseline any goal-shaping idea must beat. We left it out on purpose, because it would widen our negative result.
Feng et al. GSTransform. 2025. paper	Per-instruction views of the vectors, without re-encoding the documents.	It delivers the embed-once savings our round 5 design aimed at, without designing cards.
Qidwai et al. CAMI. 2026. paper	An agent proposes extra indexes and picks a set of them within a cost budget.	It shapes the index on the documents and a budget, not on the goal. That fits our negative result.
Microsoft Research. GraphRAG auto-tuning. 2024 [vendor]. post	Generates its extraction plan from about one percent of the documents.	Design from the documents, the form that worked for us.

Paper: Su et al. INSTRUCTOR. ACL Findings 2023. paper
What it shows: One encoder that makes task-specific vectors from an instruction.
How our results relate: It does at question time what goal-shaped cards tried to do at build time.

Paper: Weller et al. FollowIR. EMNLP 2024. paper
What it shows: A test set with two sets of human judgements per query, before and after the instruction changes. It also defines the p-MRR score.
How our results relate: Our real-data test in round 5. Re-picking cards per instruction scored p-MRR about zero.

Paper: Weller et al. Promptriever. ICLR 2025. paper
What it shows: Retrievers trained to follow instructions, as language models do.
How our results relate: The strong baseline any goal-shaping idea must beat. We left it out on purpose, because it would widen our negative result.

Paper: Feng et al. GSTransform. 2025. paper
What it shows: Per-instruction views of the vectors, without re-encoding the documents.
How our results relate: It delivers the embed-once savings our round 5 design aimed at, without designing cards.

Paper: Qidwai et al. CAMI. 2026. paper
What it shows: An agent proposes extra indexes and picks a set of them within a cost budget.
How our results relate: It shapes the index on the documents and a budget, not on the goal. That fits our negative result.

Paper: Microsoft Research. GraphRAG auto-tuning. 2024 [vendor]. post
What it shows: Generates its extraction plan from about one percent of the documents.
How our results relate: Design from the documents, the form that worked for us.

A model designs, cheap tools do the bulk work

Paper	What it shows	How our results relate
Wan et al. TnT-LLM. KDD 2024. paper	A model builds a topic list from samples and labels some examples. Embeddings plus logistic regression then reach 0.658 accuracy, against 0.655 for the frontier model used directly.	The cost basis of our approach. Also the closest published design to the document-sample step that worked.
Pham et al. TopicGPT. NAACL 2024. paper	Topics written by a model match human categories better than classic topic models.	It supports having a model name and merge the topics found.
Hsieh et al. Distilling Step-by-Step. ACL Findings 2023. paper. SetFit. 2022. library	Small models taught by large ones can beat the teacher at the task. Small few-shot models beat GPT-3 while 1,600 times smaller.	This is why the filter and the classifiers should be small models, not frontier calls.
Databricks. Scaling document classification to 100k+ labels. Jul 2026 [vendor]. post	A vector shortlist plus a small model costs about one hundredth per document of direct frontier classification. Accuracy was higher, at 0.81 against 0.76 [vendor].	A production-scale match for the two-pass constant saving.
Arora et al. EVAPORATE. VLDB 2023. paper	A model writes extraction code from a sample. The code then runs over the whole collection. Token cost falls by two orders of magnitude.	The same shape of saving, measured by others.
Sarthi et al. RAPTOR. ICLR 2024. paper	Clusters embeddings, summarises the clusters level by level, and indexes every level.	The closest cluster-then-summarise system. It uses a Gaussian mixture where we used density clustering.
Grootendorst. BERTopic. 2022. library	Density clustering over shrunk embeddings. It includes outlier reduction, topic merging, and guided or zero-shot topic lists.	The standard tool we used in rounds 2 to 5, by design, instead of a home-made version.
Edge et al. GraphRAG. 2024 (Microsoft). paper. TREX. Mar 2025. paper. LightRAG. EMNLP 2025. paper	Graph indexing that uses a model heavily, for questions about a whole collection. Indexing bills ran $51 to $389 per test collection. Lighter alternatives exist.	The cost of having a model read everything. A vector design with topics and clusters competes in this space. See the guide's knowledge-graph study.

Paper: Wan et al. TnT-LLM. KDD 2024. paper
What it shows: A model builds a topic list from samples and labels some examples. Embeddings plus logistic regression then reach 0.658 accuracy, against 0.655 for the frontier model used directly.
How our results relate: The cost basis of our approach. Also the closest published design to the document-sample step that worked.

Paper: Pham et al. TopicGPT. NAACL 2024. paper
What it shows: Topics written by a model match human categories better than classic topic models.
How our results relate: It supports having a model name and merge the topics found.

Paper: Hsieh et al. Distilling Step-by-Step. ACL Findings 2023. paper. SetFit. 2022. library
What it shows: Small models taught by large ones can beat the teacher at the task. Small few-shot models beat GPT-3 while 1,600 times smaller.
How our results relate: This is why the filter and the classifiers should be small models, not frontier calls.

Paper: Databricks. Scaling document classification to 100k+ labels. Jul 2026 [vendor]. post
What it shows: A vector shortlist plus a small model costs about one hundredth per document of direct frontier classification. Accuracy was higher, at 0.81 against 0.76 [vendor].
How our results relate: A production-scale match for the two-pass constant saving.

Paper: Arora et al. EVAPORATE. VLDB 2023. paper
What it shows: A model writes extraction code from a sample. The code then runs over the whole collection. Token cost falls by two orders of magnitude.
How our results relate: The same shape of saving, measured by others.

Paper: Sarthi et al. RAPTOR. ICLR 2024. paper
What it shows: Clusters embeddings, summarises the clusters level by level, and indexes every level.
How our results relate: The closest cluster-then-summarise system. It uses a Gaussian mixture where we used density clustering.

Paper: Grootendorst. BERTopic. 2022. library
What it shows: Density clustering over shrunk embeddings. It includes outlier reduction, topic merging, and guided or zero-shot topic lists.
How our results relate: The standard tool we used in rounds 2 to 5, by design, instead of a home-made version.

Paper: Edge et al. GraphRAG. 2024 (Microsoft). paper. TREX. Mar 2025. paper. LightRAG. EMNLP 2025. paper
What it shows: Graph indexing that uses a model heavily, for questions about a whole collection. Indexing bills ran $51 to $389 per test collection. Lighter alternatives exist.
How our results relate: The cost of having a model read everything. A vector design with topics and clusters competes in this space. See the guide's knowledge-graph study.

Variety, noise and distractors

Paper	What it shows	How our results relate
Carbonell and Goldstein. MMR. SIGIR 1998. paper	The standard rule for trading relevance against variety. It ships in most vector stores.	The harvest bought the same coverage as MMR at lambda 0.3, for half the relevance cost.
Diverse Multi-Answer Retrieval with DPPs. COLING 2022. paper. SMART-RAG. 2024. paper	Determinantal point processes for picking a varied context.	The DPP reached the most coverage and let in the most distractors (0.328).
DIVA. NAACL 2025. paper. ARAGOG. 2024. paper	Varied search wins on unclear questions with several answers. MMR adds nothing on standard single-answer questions.	It sets where variety belongs: only workloads that need coverage.
Sub-Question Coverage. Salesforce and Georgia Tech. Oct 2024. paper. Beyond Relevance: Information Coverage. ICTIR 2026. paper	Coverage predicts how complete an answer is. Coverage scores track nugget coverage, the share of key facts an answer includes.	This is why we measure with S-recall and alpha-nDCG, not recall.
Cuconasu et al. The Power of Noise. SIGIR 2024. paper. The Powerless Noise. SIGIR 2026. paper	Random documents seemed to help generation. A later paper walked the effect back.	A warning against adding off-topic material on purpose without measuring it.
Amiraz et al. The Distracting Effect. ACL 2025. paper	Answerless passages close to the topic are the most damaging additions.	This is why the pool is filtered first and the distractor rate measured. The harvest let in 0.236, against the DPP's 0.328.
Clarke et al. alpha-nDCG. SIGIR 2008. paper	The ranking score that rewards coverage and discounts repeats.	Used throughout our variety test.

Paper: Carbonell and Goldstein. MMR. SIGIR 1998. paper
What it shows: The standard rule for trading relevance against variety. It ships in most vector stores.
How our results relate: The harvest bought the same coverage as MMR at lambda 0.3, for half the relevance cost.

Paper: Diverse Multi-Answer Retrieval with DPPs. COLING 2022. paper. SMART-RAG. 2024. paper
What it shows: Determinantal point processes for picking a varied context.
How our results relate: The DPP reached the most coverage and let in the most distractors (0.328).

Paper: DIVA. NAACL 2025. paper. ARAGOG. 2024. paper
What it shows: Varied search wins on unclear questions with several answers. MMR adds nothing on standard single-answer questions.
How our results relate: It sets where variety belongs: only workloads that need coverage.

Paper: Sub-Question Coverage. Salesforce and Georgia Tech. Oct 2024. paper. Beyond Relevance: Information Coverage. ICTIR 2026. paper
What it shows: Coverage predicts how complete an answer is. Coverage scores track nugget coverage, the share of key facts an answer includes.
How our results relate: This is why we measure with S-recall and alpha-nDCG, not recall.

Paper: Cuconasu et al. The Power of Noise. SIGIR 2024. paper. The Powerless Noise. SIGIR 2026. paper
What it shows: Random documents seemed to help generation. A later paper walked the effect back.
How our results relate: A warning against adding off-topic material on purpose without measuring it.

Paper: Amiraz et al. The Distracting Effect. ACL 2025. paper
What it shows: Answerless passages close to the topic are the most damaging additions.
How our results relate: This is why the pool is filtered first and the distractor rate measured. The harvest let in 0.236, against the DPP's 0.328.

Paper: Clarke et al. alpha-nDCG. SIGIR 2008. paper
What it shows: The ranking score that rewards coverage and discounts repeats.
How our results relate: Used throughout our variety test.

Test sets and evaluation

Paper	What it shows	How our results relate
Thakur et al. BEIR. NeurIPS 2021. paper	A family of search test sets for models used without task training. SciFact comes from it.	Our test set with human judgements.
Weller et al. LIMIT dataset. 2025 (above).	The one-vector stress test.	Our test of piece count. It is also where we made our largest correction.
Es et al. RAGAS. EACL 2024. paper. RAGChecker. 2024. paper	Ways to grade generated answers without a reference answer, or in fine detail.	Our unsupported-content measure follows the same idea. It uses a direct word-level check instead of a judge.

Paper: Thakur et al. BEIR. NeurIPS 2021. paper
What it shows: A family of search test sets for models used without task training. SciFact comes from it.
How our results relate: Our test set with human judgements.

Paper: Weller et al. LIMIT dataset. 2025 (above).
What it shows: The one-vector stress test.
How our results relate: Our test of piece count. It is also where we made our largest correction.

Paper: Es et al. RAGAS. EACL 2024. paper. RAGChecker. 2024. paper
What it shows: Ways to grade generated answers without a reference answer, or in fine detail.
How our results relate: Our unsupported-content measure follows the same idea. It uses a direct word-level check instead of a judge.

Evidence and limits

This list covers the papers we read in August 2026. It is not a full survey of the field. Conference venues are as published. Vendor results are the vendor's own claims, and they are marked [vendor].

Next step

Apply these ideas to an agent on step 2 of the intelligence ladder: context. Or return to Start here.

Explore this topic

We read this work in two passes, on 2026-08-19 and 2026-08-20. The first covered multi-view indexing, variety, topic-list economics and graph against vector search. The second added instruction-following search and model-designed indexes. Every arXiv entry was checked against its abstract page on 2026-08-20. The check covered title, first author and submission date. Vendor results are marked [vendor]. Conference venues are as published.