# Build the index from the documents, not from the goal

Showing the index designer a sample of the documents helps a lot. Telling it your goal makes search worse, and more context makes it worse still. Tested five ways, ending on real data judged by people.

Author: Murali Sid (https://linkedin.com/in/muralisid)
Source: https://www.agenticarchitectureskills.com/patterns/why-not-to-condition-on-the-objective
Published: 2026-08-23
Updated: 2026-10-15
Licence: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/
Cite as: Murali Sid. Build the index from the documents, not from the goal. Agentic Architecture Skills. https://www.agenticarchitectureskills.com/patterns/why-not-to-condition-on-the-objective

---


A model can design the card types for a search index. A card type is one kind of searchable piece, such as fault logs or contracts. It seemed natural to tell the designer the goal. Legal discovery might want different cards from operations. We asked whether the goal should shape how the index is built. It should not. Showing the designer a sample of the documents helps a lot. Adding the goal, business context or a data schema made search worse every time. The more context we added, the worse it got. Where the goal really changes what is relevant, apply it at question time instead.

This decision shapes what agents can find in [knowledge and memory](/memory). It belongs to the [agent data engineering layer (R14)](/layers/r14-agent-data-engineering). It sets how the chunk and index stages of the [data-to-memory pipeline](/architecture/data-to-memory) are designed. It is the evidence for pattern 4 of the [context patterns](/patterns).

## Five tests, one answer [#five-tests-one-answer]

We ran five rounds. Each round fixed a real weakness in the one before. A few terms recur. An encoder is the model that turns text into vectors. A <Term k="topic modelling">topic model</Term> groups documents into themes automatically. A seed is the random starting point of a run. A filter keeps only material that looks relevant to the goal. Schema metadata describes the fields and tables in the data.

Two checks recur too. The first compares what the designer is shown, from nothing up to the documents plus the goal. We call these the conditioning levels. The second is a cross-over test. It asks whether card types designed for goal A serve A's questions better than types designed for goal B. It also asks the reverse.

| Round   | What the designer was given                                                                                                                                           | Data                                                 | Result                                                                                                                                                                |
| ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1       | A sample of the documents plus a narrow research goal. A plain topic model. No filter.                                                                                | Synthetic. Two goals over one collection.            | The goal made search significantly worse. No cross-over of any size.                                                                                                  |
| 2 and 3 | The same, inside a goal-filtered pool, with guided topics and model-cleaned noise. Round 3 first repaired a broken topic model.                                       | Synthetic                                            | The cross-over held at two of three filter settings on one machine. It held at zero of three on a second machine. The levels comparison changed sign across settings. |
| 4       | Business context, then schema metadata, then the narrow goal. The questions spanned six departments.                                                                  | Synthetic, 200 queries                               | Each added level made search worse, step by step. The stopping rule written in advance fired.                                                                         |
| 5       | The documents are turned into vectors once and their topics found once, with no seeding. The goal enters only when a model picks card types from those shared topics. | Synthetic, 5 topic-model seeds                       | The cross-over held at one of five seeds.                                                                                                                             |
| 5       | The same design on real news collections, with two sets of human judgements per query.                                                                                | FollowIR Core17 (20 queries) and News21 (32 queries) | Goal-picked cards were the worst setup. A plain encoder with the instruction added to the query beat every card setup. Instruction following was zero.                |

## Round 1: the goal hurts [#round-1-the-goal-hurts]

Showing the designer the documents helps a lot. Adding the goal takes some of that away. The levels vary only what the designer is shown. Every level shares one encoder, one card builder and one index. Scores are <Term k="ndcg">nDCG\@10</Term>, a 0 to 1 measure of the top ten results.

<ResultChart kind="bars" title="What the designer was shown, round 1" subtitle="nDCG@10 on synthetic workload A, 150 queries. Card types designed from the documents alone in the accent colour." categories="['Pooled, one vector', 'Generic card types (no documents, no goal)', 'Matched blind chunks', 'Document sample plus the goal', 'Document sample only']" series="[{ name: 'nDCG@10', values: [0.295, 0.449, 0.588, 0.605, 0.673] }]" emphasis="['Document sample only']" min="0.2" max="0.75" ticks="[0.2, 0.4, 0.6]" caption="Showing the designer the documents is worth +0.224 over generic card types (p=0.0001). It also beats blind chunks. Adding the research goal costs -0.068 (p=0.002). The useful signal is the documents. Published systems for designing topic lists already use that signal." source="Author-reported experiment, August 2026. The comparison and limitations are described on this page." />

The cross-over test found nothing. It gave +0.020 (p=0.35) one way and -0.002 (p=0.92) the other. Card types designed for different goals served each other's questions equally well.

We also tested a sampler in this round. It spread the designer's sample across the topics found, so minority material could not vanish. It did its job. Minority-topic coverage of the sample rose from 60 to 100 percent. But it bought nothing in search (-0.015, p=0.70). Worse, it produced a significant gain on the balanced control collection. The rules we wrote before the test said the gap should vanish there. So the effect did not come from the cause we claimed.

One comparison in this round was unsound, and we record it as such. We added the goal to the front of each query as an instruction baseline. But that encoder uses fixed prefixes and does not follow instructions. So adding the goal only diluted the query. That comparison measured dilution, not instruction following. We do not use its result anywhere.

## Rounds 2 and 3: the full design, and a fragile cross-over [#rounds-2-and-3-the-full-design-and-a-fragile-cross-over]

With the filter, guided topics and noise cleaning restored, the cross-over appeared at two of three settings on one machine. It appeared at none of three on a second machine.

Round 1 tested a stripped-down version. It had no relevance filter, no model guidance of the topic model, and no model cleaning of clustering noise. Each gap could explain the failure. The card builder had to put every passage into some card. So card types aimed at three departments had to absorb the seven they were not aimed at.

Round 2 restored all three parts. It applied the filter the same way to every setup. It scored both sets of card types over one shared pool per workload. Scoring each over its own filtered pool had let the filter carry the comparison. Correcting that halved the effect. Round 3 repaired a broken topic model, which found 3 topics against 12. Round 2's own metrics had flagged that before any fix was tried.

T\_A is the set of card types designed for goal A. T\_B is the set designed for goal B.

| Share kept by the filter | A direction (T\_A beats T\_B on A) | B direction (T\_B beats T\_A on B) | Both directions? |
| ------------------------ | ---------------------------------- | ---------------------------------- | ---------------- |
| 0.35                     | +0.070 (p=0.0001)                  | +0.066 (p=0.0029)                  | Yes              |
| 0.50                     | +0.114 (p=0.0001)                  | +0.035 (p=0.0153)                  | Yes              |
| 0.75                     | +0.156 (p=0.0001)                  | +0.006 (p=0.59)                    | No               |

The A direction holds everywhere and grows as the filter loosens. The B direction is fragile. We replicated it on a second machine at the same code version. The encoder ran on the CPU with a fixed hash seed. The B direction then read -0.018 (p=0.32), +0.007 (p=0.65) and -0.040 (p=0.034, the wrong direction). That is zero of three.

We understand why. The steps that shrink and cluster the vectors give different topics on a different processor type. The designer then sees different samples and writes different card types. The B effect depends on which card types it happens to write. The conditioning levels over the same settings read +0.019, -0.061 (p=0.0018) and -0.012. They changed sign across settings. So that single-workload comparison cannot be quoted either way.

What rounds 2 and 3 showed is narrower than the claim. Goal-based design produces genuinely different card types. In one direction, each set serves its own questions better. It did not show that goal-designed card types beat good document-designed ones on any workload.

## Round 4: more context, worse search [#round-4-more-context-worse-search]

Business context and schema metadata are the signals an enterprise really has. They made search worse in proportion to how much we supplied.

Our argument for this round was simple. Enterprises do not organise data for one question. They organise a business unit's data so several agents can ask many questions. In practice the signal on hand is standing business context plus schema metadata, not a narrow goal. So we built the workload to match: 200 queries across six departments.

We wrote the rules down before building anything. One was a stopping rule. If context plus metadata did not beat the document sample alone, we would stop pursuing the claim.

<ResultChart kind="bars" title="What the designer was shown, on questions spanning six departments" subtitle="nDCG@10, 200 queries. Each level adds to the one before. Document sample only in the accent colour." categories="['Pooled, one vector', 'Matched blind chunks', 'Document sample only (30 documents spread across topics)', 'Plus business context', 'Plus schema metadata', 'Plus the narrow goal']" series="[{ name: 'nDCG@10', values: [0.303, 0.489, 0.562, 0.515, 0.494, 0.457] }]" emphasis="['Document sample only (30 documents spread across topics)']" min="0.25" max="0.62" ticks="[0.3, 0.4, 0.5, 0.6]" caption="Context plus metadata against the sample alone: -0.068, 95 percent interval -0.090 to -0.046, p=0.0001. Business context alone: -0.047 (p=0.0001). The narrow goal on top: a further -0.037 (p=0.018). The harm grows with the amount of context supplied." source="Author-reported experiment, August 2026. The comparison and limitations are described on this page." />

The card types show why. From the documents alone, the designer derived six types that followed the data. They were fault logs, system maintenance, financial risk, legal compliance, logistics and supply, and IT incidents. Given the business story, it produced four tidy types organised around the business. They were commercial agreements, operations and logistics, finance reporting, and risk and compliance.

The business framing covers the documents less well. On questions spanning six departments, coverage is what matters. The two sets were genuinely different, with a mean pairwise similarity of 0.833. So our rule for designs that converge did not fire. They were also genuinely worse. The stopping rule fired, and we stopped pursuing the claim in that form.

## Round 5: the goal enters late, on real data [#round-5-the-goal-enters-late-on-real-data]

The practical form of the idea embeds the documents once, turning them into vectors. Then a model picks card types per goal from shared topics. It does not carry the effect. On data judged by people, it is the worst setup.

Our final design separated two things that rounds 2 to 4 had mixed together. One is shaping the topic model with the goal, through seed topics written from it. The other is shaping the card design with the goal. In round 5 the documents are embedded once. Their topics are found once, with no seeding. That gives one shared set of topics that reflects the documents alone. The goal enters only later. A model picks and combines card types from those topics, and the goal also sets the filter. The cost appeal is real. A new goal then costs a few model calls, not a new embedding pass.

For this round we chose not to write the full rules down in advance. We kept it strict in other ways. Decision rules were fixed in code before any number was read. Every verdict had to hold across several topic-model seeds. Every setup was reported, including those that beat the method.

On the synthetic data we used the validated card builder and five topic-model seeds. The two-way cross-over held at one seed of five. The A direction was firmly positive, at +0.034 to +0.099, and significant in four of five. The B direction ranged from -0.038 to +0.025. The round 3 cross-over had come from the goal-written seed topics, not from the idea. The practical embed-once version does not carry it.

FollowIR supplies the real test. It holds news collections where each query comes with an instruction. Each query has two sets of human judgements, before and after the instruction changes. So an index that follows the goal can be told apart from one that ignores it. We scored five setups.

* One pooled vector per document.
* The same encoder with the instruction added to the front of the query. This is a weak <Term k="instruction-following retriever">instruction-following baseline</Term>, because that encoder was not trained to follow instructions.
* Matched blind chunks.
* One goal-blind set of card types, derived from the documents.
* The card types a model picked per instruction from the shared topics.

<ResultChart
  kind="bars"
  title="FollowIR: five setups on news collections judged by people"
  subtitle="nDCG@10 under each query's original judgements. Core17: 20 queries, seed 13, the validated card format. News21: 32 queries, mean of two seeds for the shared topics."
  categories="['Pooled, one vector', 'Instruction added to the query', 'Matched blind chunks', 'Card types from the documents, goal-blind', 'Card types picked per goal']"
  series="[
  { name: 'Core17', color: 'slate', values: [0.393, 0.394, 0.403, 0.409, 0.349] },
  { name: 'News21', color: 'ember', values: [0.386, 0.416, 0.364, 0.327, 0.331] },
]"
  min="0.3"
  max="0.45"
  ticks="[0.3, 0.35, 0.4, 0.45]"
  caption="On Core17 the goal-picked cards are the worst setup, 0.06 below the document-derived card types. A confirmation run with the fully validated card format widened the gap rather than closing it. On News21 the instruction-added encoder is the best setup, and both card setups score lowest. Under the changed instructions the order is the same. The instruction-following score p-MRR is about zero for both instruction-aware setups. Neither follows the instruction."
  source="Author-reported experiment, August 2026. The comparison and limitations are described on this page."
/>

<Term k="p-mrr">p-MRR</Term> scores whether a ranking changes the right way when the instruction changes. Zero means it does not change.

We recorded two caveats with the result. Both point the same way. The encoder is a weak instruction baseline. A retriever trained to follow instructions would widen the gap against the method, not narrow it. And we capped the candidate pool depth and document length for speed, equally for every setup.

We did not run a third collection, Robust04. The direction was already consistent across the synthetic test and two real collections. It would have added support, not new information.

## A worked example [#a-worked-example]

Take a utility's maintenance archive used by a legal team and an operations team. Build one set of card types from a sample of the archive. Do not build one index for legal and another for operations. If the legal team's instructions change what counts as relevant, apply them at question time. Use a retriever trained to follow instructions. This example applies the finding. We tested synthetic data and news, not a maintenance archive.

## What we conclude [#what-we-conclude]

<PrincipleCard eyebrow="Verdict" title="The goal is the wrong input for building the index" kind="risk">
  Shaping the card design with the goal, in any of five forms, does not improve search. It usually harms it. The harm was -0.068 in round 1. It was -0.047 to -0.068 at each level of context in round 4. It was about -0.06 on real data judged by people in round 5. The cross-over that briefly appeared came from the topic seeds. It did not repeat across machines or seeds. Where the goal must shape search, a retriever trained to follow instructions is the established, cheaper tool. Even an untrained encoder with the instruction added beat every card setup on one real collection. The one lasting positive is that document-derived card types stay competitive with chunks and pooling. That is not new.
</PrincipleCard>

## Evidence and limits [#evidence-and-limits]

<EvidenceBadge status="author-measured" /> Synthetic results across seeds and machines. FollowIR with human judgements.

We ran every round ourselves, on two machines. Nobody else has repeated them yet. The FollowIR collections are small, at 20 and 32 queries. The encoder was small and not trained to follow instructions.

## Next step [#next-step]

Next, see [where the cost saving really comes from](/patterns/gate-and-economics) when a collection is large. Or go back to the [nine context patterns](/patterns).

<details>
  <summary>
    Explore this topic
  </summary>

  * Weller et al., FollowIR, EMNLP 2024, for the collections and the p-MRR score. [https://arxiv.org/abs/2403.15246](https://arxiv.org/abs/2403.15246)
  * Promptriever (ICLR 2025), INSTRUCTOR (ACL Findings 2023), GSTransform (2025), CAMI (2026) and TnT-LLM (KDD 2024). This is the earlier published work that already covers this idea. All are on the [reading list](/patterns/reading-list).
  * [How to test a context design](/patterns/how-to-test-a-context-design), for the rules written in advance, the round without them, and the second-machine rerun.
  * The rounds ran in August 2026.
</details>