# On real scientific text, plain chunks beat purpose cards

On 5,183 scientific abstracts judged by people, purpose cards score below plain chunks of the same count. Cards designed by a model do no better. Adding keyword search gives the best result.

Author: Murali Sid (https://linkedin.com/in/muralisid)
Source: https://www.agenticarchitectureskills.com/patterns/real-prose
Published: 2026-08-23
Updated: 2026-10-15
Licence: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/
Cite as: Murali Sid. On real scientific text, plain chunks beat purpose cards. Agentic Architecture Skills. https://www.agenticarchitectureskills.com/patterns/real-prose

---


Our [first test](/patterns/aspect-dilution) used made-up documents with clearly separate topics, and questions about one topic. Real writing is rarely that tidy. So we asked whether purpose cards still beat plain chunks on real text, judged by people. They do not. On scientific abstracts, cards score below plain chunks of the same count. Cards designed by a model do no better. The best result comes from adding keyword search.

A purpose card is one searchable piece per topic in a document, with its own vector. Plain chunks are the same text cut into fixed-size pieces, with no regard to meaning. Both are ways to prepare documents for [knowledge and memory](/memory). The choice belongs to the [agent data engineering layer (R14)](/layers/r14-agent-data-engineering), at the chunk stage of the [data-to-memory pipeline](/architecture/data-to-memory).

## The test: real abstracts, real judgements [#the-test-real-abstracts-real-judgements]

SciFact holds 5,183 scientific abstracts and 300 claims. People judged which abstracts are relevant to each claim, so no model labels anything. An abstract moves through background, method, result and implication. A claim usually concerns one of those parts. Unlike LIMIT, a public test that comes already split by topic, the boundaries here are not marked. The card builder has to find them. The card types have to be designed, and they may be wrong. That is the real situation the pattern is meant for.

We compared the same setups as before. All are scored per document, with the same encoder.

* Keyword search (BM25).
* One <Term k="pooled embedding">pooled vector</Term> per document.
* <Term k="matched control">Matched blind chunks</Term>, 2.7 per document.
* Purpose cards, 2.8 per document. They use hand-written card types for the roles in an abstract.
* The cards fused with keyword search by reciprocal rank. Fusing merges the two ranked lists.

<ResultChart kind="bars" title="SciFact: purpose cards against matched chunks, judged by people" subtitle="nDCG@10 over 300 claims, seed 13. Card setups in the accent colour." categories="['BM25 keyword search', 'Meaning-based, pooled', 'Matched blind chunks (2.7 per document)', 'Purpose cards, hand-written types (2.8 per document)', 'Purpose cards, model-designed types (2.5 per document)', 'Cards fused with BM25']" series="[{ name: 'nDCG@10', values: [0.662, 0.645, 0.667, 0.635, 0.626, 0.685] }]" emphasis="['Purpose cards, hand-written types (2.8 per document)', 'Purpose cards, model-designed types (2.5 per document)']" min="0.55" max="0.72" ticks="[0.55, 0.6, 0.65, 0.7]" caption="The cards are the two lowest meaning-based setups. The hybrid is the best setup. Its keyword half does real work on text full of gene names, drug names and measured quantities." source="Author-reported experiment, August 2026. The comparison and limitations are described on this page." />

Purpose cards score 0.635. Matched chunks score 0.667. The gap is -0.032, with a 95 percent interval of -0.057 to -0.008 and p=0.0115. The cards lost on 70 claims, won on 36 and tied on 194. Giving both setups the same number of vectors barely moved the gap. So piece count is not the reason. The cards are simply worse here.

Fusing cards with BM25 gives 0.685. That is +0.040 over the pooled vector (p=0.012), and the best result in the table. The gain comes from the keyword side.

## A defect we caught and fixed [#a-defect-we-caught-and-fixed]

The first version of this test reported -0.193 for the cards. The number was real, but a defect caused it. The card builder kept only the top two passages per topic. So the cards held 44.6 percent of each document, while the chunks held 100 percent. A card set holding less than half the text cannot be fairly compared with one holding all of it.

We kept that broken setup in the results, at 0.452, as a lesson. How much of the source text each setup keeps is part of a fair comparison. Measure it before believing a result. The corrected cards cover the whole document and score 0.635.

## Can a model design better card types? [#can-a-model-design-better-card-types]

After the first result, we suspected the hand-written card types were the problem. They followed the roles in an abstract, not its content. That is exactly the mismatch the method warns about. Perhaps a model shown the documents and the questions would design better ones. We tested that directly.

Before any scoring, one model call saw a sample of documents and 809 claims from the training split. It was asked for four to seven card types, each with anchor phrases. An anchor phrase is a short description that decides which passages go on a card. The model never saw the 300 test claims. It proposed sensible content types: molecular mechanisms, disease pathology, experimental methods, physiological systems and clinical outcomes. The same card builder then ran with those types.

The model-designed cards score 0.626.

* Against the hand-written card types: -0.009 (p=0.48). A tie.
* Against the pooled vector: -0.019 (p=0.11).
* Against matched chunks: -0.042, with a 95 percent interval of -0.069 to -0.016 and p=0.0018.

We reran it on a second machine with a fixed hash seed, a setting that controls random order. That changed the training sample the model saw, so it wrote different card types. The result held: -0.044 against chunks, p=0.004.

## How a card boundary splits the evidence [#how-a-card-boundary-splits-the-evidence]

In the synthetic test, a question targets one topic. Splitting by topic puts the whole answer on one card. A SciFact claim concerns a finding. The evidence for a finding is spread across the method, the result and the implication. A card boundary separates them. A blind window keeps together whichever parts happen to sit side by side.

Neither setup matches what the question asks about. The one that keeps more nearby text wins. No set of card types helps here, whether a model or a person designs it. This marks the edge of where purpose cards work.

## A worked example [#a-worked-example]

Clinical notes in healthcare have a similar shape. A question about a treatment outcome often needs the method and the result together. Split them onto separate cards and the match may weaken. We have not tested clinical notes. The safe default is plain chunks plus keyword search, and cards only if they win on your own questions.

## What we conclude [#what-we-conclude]

<PrincipleCard eyebrow="Verdict" title="Where purpose cards lose, and the rule that follows" kind="control">
  When questions concern whole findings rather than one topic, purpose cards score below matched chunks. The gap is -0.032 for hand-written types and -0.042 for model-designed types. Both are significant. No card design closes it. The rule: purpose cards must beat matched chunks on your own questions before you adopt them. Keep keyword search either way. It produced the best result.
</PrincipleCard>

## Evidence and limits [#evidence-and-limits]

<EvidenceBadge status="author-measured" /> Relevance was judged by people, with no model labels. We reran it on a second machine. Nobody else has repeated it yet.

SciFact is one kind of real text, in English. Its abstracts are short. We have not tested long documents with clear sections, such as manuals or contracts.

## Next step [#next-step]

See whether telling the index builder your goal helps, in [build the index from the documents, not the goal](/patterns/why-not-to-condition-on-the-objective). Or go back to the [nine context patterns](/patterns).

<details>
  <summary>
    Explore this topic
  </summary>

  * BEIR, the benchmark family SciFact belongs to: Thakur et al., NeurIPS 2021. [https://arxiv.org/abs/2104.08663](https://arxiv.org/abs/2104.08663)
  * Weller et al., When do Generative Query and Document Expansions Fail?, EACL 2024. It shows derived text in an index can hurt strong retrievers. [https://arxiv.org/abs/2309.08541](https://arxiv.org/abs/2309.08541)
  * [The nine context patterns](/patterns), where this boundary becomes a rule.
  * The experiment ran in August 2026.
</details>