On real scientific text, plain chunks beat purpose cards

Our first test used made-up documents with clearly separate topics, and questions about one topic. Real writing is rarely that tidy. So we asked whether purpose cards still beat plain chunks on real text, judged by people. They do not. On scientific abstracts, cards score below plain chunks of the same count. Cards designed by a model do no better. The best result comes from adding keyword search.

A purpose card is one searchable piece per topic in a document, with its own vector. Plain chunks are the same text cut into fixed-size pieces, with no regard to meaning. Both are ways to prepare documents for knowledge and memory. The choice belongs to the agent data engineering layer (R14), at the chunk stage of the data-to-memory pipeline.

The test: real abstracts, real judgements

SciFact holds 5,183 scientific abstracts and 300 claims. People judged which abstracts are relevant to each claim, so no model labels anything. An abstract moves through background, method, result and implication. A claim usually concerns one of those parts. Unlike LIMIT, a public test that comes already split by topic, the boundaries here are not marked. The card builder has to find them. The card types have to be designed, and they may be wrong. That is the real situation the pattern is meant for.

We compared the same setups as before. All are scored per document, with the same encoder.

• Keyword search (BM25).
• One pooled vectorPooled (single-vector) embedding. One numeric fingerprint for a whole document. Cheap and standard, but for a document about several things it is an average of all of them, so it matches each of them weakly. per document.
• Matched blind chunksMatched control. A comparison arm given the same budget as the method under test, here the same number of embeddings per document, so the experiment measures the idea rather than the extra capacity., 2.7 per document.
• Purpose cards, 2.8 per document. They use hand-written card types for the roles in an abstract.
• The cards fused with keyword search by reciprocal rank. Fusing merges the two ranked lists.

Purpose cards score 0.635. Matched chunks score 0.667. The gap is -0.032, with a 95 percent interval of -0.057 to -0.008 and p=0.0115. The cards lost on 70 claims, won on 36 and tied on 194. Giving both setups the same number of vectors barely moved the gap. So piece count is not the reason. The cards are simply worse here.

Fusing cards with BM25 gives 0.685. That is +0.040 over the pooled vector (p=0.012), and the best result in the table. The gain comes from the keyword side.

A defect we caught and fixed

The first version of this test reported -0.193 for the cards. The number was real, but a defect caused it. The card builder kept only the top two passages per topic. So the cards held 44.6 percent of each document, while the chunks held 100 percent. A card set holding less than half the text cannot be fairly compared with one holding all of it.

We kept that broken setup in the results, at 0.452, as a lesson. How much of the source text each setup keeps is part of a fair comparison. Measure it before believing a result. The corrected cards cover the whole document and score 0.635.

Can a model design better card types?

After the first result, we suspected the hand-written card types were the problem. They followed the roles in an abstract, not its content. That is exactly the mismatch the method warns about. Perhaps a model shown the documents and the questions would design better ones. We tested that directly.

Before any scoring, one model call saw a sample of documents and 809 claims from the training split. It was asked for four to seven card types, each with anchor phrases. An anchor phrase is a short description that decides which passages go on a card. The model never saw the 300 test claims. It proposed sensible content types: molecular mechanisms, disease pathology, experimental methods, physiological systems and clinical outcomes. The same card builder then ran with those types.

The model-designed cards score 0.626.

• Against the hand-written card types: -0.009 (p=0.48). A tie.
• Against the pooled vector: -0.019 (p=0.11).
• Against matched chunks: -0.042, with a 95 percent interval of -0.069 to -0.016 and p=0.0018.

We reran it on a second machine with a fixed hash seed, a setting that controls random order. That changed the training sample the model saw, so it wrote different card types. The result held: -0.044 against chunks, p=0.004.

How a card boundary splits the evidence

In the synthetic test, a question targets one topic. Splitting by topic puts the whole answer on one card. A SciFact claim concerns a finding. The evidence for a finding is spread across the method, the result and the implication. A card boundary separates them. A blind window keeps together whichever parts happen to sit side by side.

Neither setup matches what the question asks about. The one that keeps more nearby text wins. No set of card types helps here, whether a model or a person designs it. This marks the edge of where purpose cards work.

A worked example

Clinical notes in healthcare have a similar shape. A question about a treatment outcome often needs the method and the result together. Split them onto separate cards and the match may weaken. We have not tested clinical notes. The safe default is plain chunks plus keyword search, and cards only if they win on your own questions.

What we conclude

Evidence and limits

Measured by the author Relevance was judged by people, with no model labels. We reran it on a second machine. Nobody else has repeated it yet.

SciFact is one kind of real text, in English. Its abstracts are short. We have not tested long documents with clear sections, such as manuals or contracts.

Next step

See whether telling the index builder your goal helps, in build the index from the documents, not the goal. Or go back to the nine context patterns.