Build the index from the documents, not from the goal

A model can design the card types for a search index. A card type is one kind of searchable piece, such as fault logs or contracts. It seemed natural to tell the designer the goal. Legal discovery might want different cards from operations. We asked whether the goal should shape how the index is built. It should not. Showing the designer a sample of the documents helps a lot. Adding the goal, business context or a data schema made search worse every time. The more context we added, the worse it got. Where the goal really changes what is relevant, apply it at question time instead.

This decision shapes what agents can find in knowledge and memory. It belongs to the agent data engineering layer (R14). It sets how the chunk and index stages of the data-to-memory pipeline are designed. It is the evidence for pattern 4 of the context patterns.

Five tests, one answer

We ran five rounds. Each round fixed a real weakness in the one before. A few terms recur. An encoder is the model that turns text into vectors. A topic modelTopic modelling. Grouping documents into themes automatically by clustering their embeddings, then naming each group. Here a model names and merges the groups from keyword lists; it never reads every document. groups documents into themes automatically. A seed is the random starting point of a run. A filter keeps only material that looks relevant to the goal. Schema metadata describes the fields and tables in the data.

Two checks recur too. The first compares what the designer is shown, from nothing up to the documents plus the goal. We call these the conditioning levels. The second is a cross-over test. It asks whether card types designed for goal A serve A's questions better than types designed for goal B. It also asks the reverse.

Round 1: the goal hurts

Showing the designer the documents helps a lot. Adding the goal takes some of that away. The levels vary only what the designer is shown. Every level shares one encoder, one card builder and one index. Scores are nDCG@10nDCG@10. A 0 to 1 score of how good the first ten search results are. It rewards putting the right documents near the top and counts every relevant document the judges marked. A difference of 0.03 is small but real on a few hundred queries; 0.2 is large., a 0 to 1 measure of the top ten results.

The cross-over test found nothing. It gave +0.020 (p=0.35) one way and -0.002 (p=0.92) the other. Card types designed for different goals served each other's questions equally well.

We also tested a sampler in this round. It spread the designer's sample across the topics found, so minority material could not vanish. It did its job. Minority-topic coverage of the sample rose from 60 to 100 percent. But it bought nothing in search (-0.015, p=0.70). Worse, it produced a significant gain on the balanced control collection. The rules we wrote before the test said the gap should vanish there. So the effect did not come from the cause we claimed.

One comparison in this round was unsound, and we record it as such. We added the goal to the front of each query as an instruction baseline. But that encoder uses fixed prefixes and does not follow instructions. So adding the goal only diluted the query. That comparison measured dilution, not instruction following. We do not use its result anywhere.

Rounds 2 and 3: the full design, and a fragile cross-over

With the filter, guided topics and noise cleaning restored, the cross-over appeared at two of three settings on one machine. It appeared at none of three on a second machine.

Round 1 tested a stripped-down version. It had no relevance filter, no model guidance of the topic model, and no model cleaning of clustering noise. Each gap could explain the failure. The card builder had to put every passage into some card. So card types aimed at three departments had to absorb the seven they were not aimed at.

Round 2 restored all three parts. It applied the filter the same way to every setup. It scored both sets of card types over one shared pool per workload. Scoring each over its own filtered pool had let the filter carry the comparison. Correcting that halved the effect. Round 3 repaired a broken topic model, which found 3 topics against 12. Round 2's own metrics had flagged that before any fix was tried.

T_A is the set of card types designed for goal A. T_B is the set designed for goal B.

The A direction holds everywhere and grows as the filter loosens. The B direction is fragile. We replicated it on a second machine at the same code version. The encoder ran on the CPU with a fixed hash seed. The B direction then read -0.018 (p=0.32), +0.007 (p=0.65) and -0.040 (p=0.034, the wrong direction). That is zero of three.

We understand why. The steps that shrink and cluster the vectors give different topics on a different processor type. The designer then sees different samples and writes different card types. The B effect depends on which card types it happens to write. The conditioning levels over the same settings read +0.019, -0.061 (p=0.0018) and -0.012. They changed sign across settings. So that single-workload comparison cannot be quoted either way.

What rounds 2 and 3 showed is narrower than the claim. Goal-based design produces genuinely different card types. In one direction, each set serves its own questions better. It did not show that goal-designed card types beat good document-designed ones on any workload.

Round 4: more context, worse search

Business context and schema metadata are the signals an enterprise really has. They made search worse in proportion to how much we supplied.

Our argument for this round was simple. Enterprises do not organise data for one question. They organise a business unit's data so several agents can ask many questions. In practice the signal on hand is standing business context plus schema metadata, not a narrow goal. So we built the workload to match: 200 queries across six departments.

We wrote the rules down before building anything. One was a stopping rule. If context plus metadata did not beat the document sample alone, we would stop pursuing the claim.

The card types show why. From the documents alone, the designer derived six types that followed the data. They were fault logs, system maintenance, financial risk, legal compliance, logistics and supply, and IT incidents. Given the business story, it produced four tidy types organised around the business. They were commercial agreements, operations and logistics, finance reporting, and risk and compliance.

The business framing covers the documents less well. On questions spanning six departments, coverage is what matters. The two sets were genuinely different, with a mean pairwise similarity of 0.833. So our rule for designs that converge did not fire. They were also genuinely worse. The stopping rule fired, and we stopped pursuing the claim in that form.

Round 5: the goal enters late, on real data

The practical form of the idea embeds the documents once, turning them into vectors. Then a model picks card types per goal from shared topics. It does not carry the effect. On data judged by people, it is the worst setup.

Our final design separated two things that rounds 2 to 4 had mixed together. One is shaping the topic model with the goal, through seed topics written from it. The other is shaping the card design with the goal. In round 5 the documents are embedded once. Their topics are found once, with no seeding. That gives one shared set of topics that reflects the documents alone. The goal enters only later. A model picks and combines card types from those topics, and the goal also sets the filter. The cost appeal is real. A new goal then costs a few model calls, not a new embedding pass.

For this round we chose not to write the full rules down in advance. We kept it strict in other ways. Decision rules were fixed in code before any number was read. Every verdict had to hold across several topic-model seeds. Every setup was reported, including those that beat the method.

On the synthetic data we used the validated card builder and five topic-model seeds. The two-way cross-over held at one seed of five. The A direction was firmly positive, at +0.034 to +0.099, and significant in four of five. The B direction ranged from -0.038 to +0.025. The round 3 cross-over had come from the goal-written seed topics, not from the idea. The practical embed-once version does not carry it.

FollowIR supplies the real test. It holds news collections where each query comes with an instruction. Each query has two sets of human judgements, before and after the instruction changes. So an index that follows the goal can be told apart from one that ignores it. We scored five setups.

• One pooled vector per document.
• The same encoder with the instruction added to the front of the query. This is a weak instruction-following baselineInstruction-following retriever. An embedding model trained so that a stated instruction (for example, only reports from outside the UK) changes what it ranks as relevant, instead of only the query words., because that encoder was not trained to follow instructions.
• Matched blind chunks.
• One goal-blind set of card types, derived from the documents.
• The card types a model picked per instruction from the shared topics.

p-MRRp-MRR (instruction following score). A FollowIR benchmark score of whether a search system changes its ranking the right way when the instruction changes. Zero means the system ignores the instruction. scores whether a ranking changes the right way when the instruction changes. Zero means it does not change.

We recorded two caveats with the result. Both point the same way. The encoder is a weak instruction baseline. A retriever trained to follow instructions would widen the gap against the method, not narrow it. And we capped the candidate pool depth and document length for speed, equally for every setup.

We did not run a third collection, Robust04. The direction was already consistent across the synthetic test and two real collections. It would have added support, not new information.

A worked example

Take a utility's maintenance archive used by a legal team and an operations team. Build one set of card types from a sample of the archive. Do not build one index for legal and another for operations. If the legal team's instructions change what counts as relevant, apply them at question time. Use a retriever trained to follow instructions. This example applies the finding. We tested synthetic data and news, not a maintenance archive.

What we conclude

Evidence and limits

Measured by the author Synthetic results across seeds and machines. FollowIR with human judgements.

We ran every round ourselves, on two machines. Nobody else has repeated them yet. The FollowIR collections are small, at 20 and 32 queries. The encoder was small and not trained to follow instructions.

Next step

Next, see where the cost saving really comes from when a collection is large. Or go back to the nine context patterns.