Choosing varied results, and whether the reader matters

After a search finds candidates, something must pick the ten that reach the agent. The ten most similar often repeat the same point. Ten deliberately different ones cover more ground, but may include passages that do not answer the question. We asked which selection rule gives the best trade. The answer: keep the most relevant results at the top, then fill the last places with the least similar candidates. We call this the harvest. It matched the coverage of the standard method, MMR, at half the relevance cost. It also let in fewer junk passages. We also tested whether a model reader needs more variety than a person. That idea did not survive a fair judge.

This choice is the last step before material enters the agent's context. It is part of how agents use knowledge and memory. It sits at the retrieve stage of the data-to-memory pipeline, which the agent data engineering layer (R14) builds.

The test: questions where the sub-topics are known

Each of 120 queries has several distinct facts. Each candidate document states at most one of them. So we can count exactly which sub-topics a selection covers. We did not find sub-topics by clustering. That would have been circular, because one rule under test selects by cluster.

First, candidates are cut to the most relevant half, as a first-pass filter would do. Then a pool of answerless documents is mixed in. We call these distractors. Each rule then selects ten.

Three scores tell the rules apart. S-recallCoverage metrics (S-recall, alpha-nDCG). Scores that reward a result list for covering different sub-topics rather than repeating the same one. S-recall is the share of known sub-topics that appear in the top ten; alpha-nDCG discounts repeats. is the share of known sub-topics covered. nDCG@10nDCG@10. A 0 to 1 score of how good the first ten search results are. It rewards putting the right documents near the top and counts every relevant document the judges marked. A difference of 0.03 is small but real on a few hundred queries; 0.2 is large. measures relevance. The distractor rate is the share of answerless documents let in.

We tested seven rules.

• Relevance only: the ten most relevant.
• MMRDiversity selection (MMR, DPP, harvest). Rules for choosing which ten candidates to pass on so that they cover different sub-topics rather than repeating the top hit. MMR and DPP re-rank the whole list; the harvest keeps the most relevant head and spends a fixed quota on the least similar tail. (maximal marginal relevance), at lambda 0.7 and 0.3. It is the standard rule, built into most vector stores. It trades relevance for variety across the whole list. A lower lambda means more variety.
• DPP (determinantal point process), a maths method that picks a spread-out set.
• Cluster round-robin: one result from each cluster in turn.
• The harvest, at 20 and 40 percent. Keep the most relevant results, then fill that share of the ten places with the least similar candidates.

The two harvest settings sit above and to the right of MMR and cluster round-robin at similar coverage. The DPP reaches the most coverage and pays the most relevance for it.

Table view

Point	Relevance (nDCG@10)	Sub-topic coverage (S-recall)
Relevance only	0.827	0.372
MMR, lambda 0.7	0.802	0.405
Harvest 20%	0.808	0.567
Cluster round-robin	0.774	0.545
Harvest 40%	0.781	0.686
MMR, lambda 0.3	0.729	0.686
DPP	0.699	0.746

Source: Author-reported experiment, August 2026. The comparison and limitations are described on this page.

Alpha-nDCG is a ranking score that rewards coverage and discounts repeats. Read along the rows. The 40 percent harvest and MMR at lambda 0.3 reach the same coverage, 0.686. Against relevance-only ranking that is a gain of 0.315 for the harvest and 0.314 for MMR, from the unrounded scores. The table shows rounded values. The harvest pays 0.046 of relevance for it. MMR pays 0.098. The harvest also lets in fewer distractors: 0.236, against 0.285 for MMR and 0.328 for the DPP.

The reason shows in how the rules work. MMR and the DPP spread out the whole list, so even the top drifts away from relevance. The harvest keeps a relevant top and spends only the tail. All coverage gains over relevance-only ranking are significant at p=0.0001 after Holm correction. So are all the relevance costs.

An earlier version of this test reported coverage as nearly free. It was wrong. A coding bug put two distractors in each task instead of forty. So nothing could be lost, and every relevance score sat at its maximum. The corrected trade is the one above.

The reader test: does the right amount of variety depend on who reads?

We expected it would. The effect shrank to nothing under a fair judge.

The pool, the queries and the writing model stayed the same throughout. Only the selection rule and the reader changed. For a model reader, the selection is evidence for a model. It must write an account of everything the material covers. For a person, the same selection becomes a short brief they will read once and act on. Sixty tasks, two rules (relevance and harvest) and two readers give 240 generated texts.

We predicted a difference between readers, not a gain for everyone. The harvest should raise coverage more for the model reader than for the person.

Three judges scored coverage of the known sub-topics in each text. A judge is a model that grades another model's output. One judge came from the same company as the writing model. One was a larger model from that company. One came from a different company.

+0.071 (p=0.009), +0.047 (p=0.211), +0.008 (p=0.763). The estimate shrinks as the judge moves away from the writer's own company. It vanishes with the judge from another company. That pattern points to a flaw in the judging, not a real effect. It is not disproved. All three estimates are positive, and 60 tasks cannot resolve an effect this small. But no evidence can be cited for it.

Table view

Category	Reader difference	95% interval
Same-company judge, small	+0.071	+0.022 to +0.125
Same-company judge, large	+0.047	−0.024 to +0.117
Different-company judge	+0.008	−0.039 to +0.056

Source: Author-reported experiment, August 2026. The comparison and limitations are described on this page.

One finding from the same study needs no judge, and it survived every check. We counted words in each output that no selected document supports. Harvesting reduced this unsupported content.

• For the brief for a person: -0.067 (p=0.0018).
• For the account written for a model: -0.048 (p=0.067).

A selection that covers more of the material gives the writer less reason to invent. We did not predict this. It does not depend on a judge. It is the most solid result of the reader study.

A worked example

A utility planner asks an agent for every known cause of a substation fault. That question has several answers, so a varied set of ten helps. The harvest keeps the strongest matches at the top and adds different causes below them. A question with one answer, such as the date of the last inspection, gains nothing from variety. This example shows how to apply the rule. We tested it on built questions, not on utility records.

What we conclude

Evidence and limits

Mixed evidence The selection half is measured on built ground truth. The reader half depends on model judges and is unresolved.

The queries were built for the test, not taken from real users. Sixty tasks are too few to detect a small reader effect. We ran this ourselves. Nobody else has repeated it yet.

Next step

See how all these tests were kept honest, in how to test a context design. Or go back to the nine context patterns.

Explore this topic

• Carbonell and Goldstein, MMR, SIGIR 1998. Diverse multi-answer retrieval with determinantal point processes, COLING 2022. DIVA, NAACL 2025. The Distracting Effect, ACL 2025. The Power of Noise (SIGIR 2024) and its retraction (SIGIR 2026). All are on the reading list.
• Alpha-nDCG, Clarke et al., SIGIR 2008. Sub-question coverage, Salesforce and Georgia Tech, Oct 2024. Both explain why coverage needs coverage scores, not recall.
• The experiment ran in August 2026.