# How to test a context design
Give the plain method the same budget as the clever one, and make sure the test can fail. Then write the rules first, let someone try to break the result, and rerun it on a second machine. Five of our own headline results reversed, and two causes explain all five.
Author: Murali Sid (https://linkedin.com/in/muralisid)
Source: https://www.agenticarchitectureskills.com/patterns/how-to-test-a-context-design
Published: 2026-08-23
Updated: 2026-10-15
Licence: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/
Cite as: Murali Sid. How to test a context design. Agentic Architecture Skills. https://www.agenticarchitectureskills.com/patterns/how-to-test-a-context-design
---
Experiments that confirm what their authors hoped for are easy to produce and unsafe to rely on. We wanted to know how to make a search result hard to fool. Five of our own headline results reversed at least once. Two causes explain all five. Either the control did not get the same budget as the method, or the measurement could not come out negative. So give the plain method the same budget, and make sure the test can fail. The steps below show how we did both, and what they caught.
Use this method before you change how documents are prepared for [knowledge and memory](/memory). It covers the chunk, embed and retrieve stages of the [data-to-memory pipeline](/architecture/data-to-memory). That work belongs to the [agent data engineering layer (R14)](/layers/r14-agent-data-engineering). It is the evidence for pattern 9 of the [context patterns](/patterns). The general approach behind it is on [how we evaluate evidence](/research/method).
## Step 1: give the control the same budget [#step-1-give-the-control-the-same-budget]
Search tools turn text into vectors, lists of numbers that stand for meaning. Every test compares three setups.
* One pooled vector per document, a single vector for the whole text.
* The same text cut into blind fixed-size windows. The window width is set so each document gets as many windows as it gets cards.
* The same text split by purpose into cards, one searchable piece per topic.
The gap between the first two shows what more vectors buy. The gap between the last two is the idea under test. Matching the number of pieces per document between chunks and cards is what makes the comparison fair. Getting that wrong was our single largest error.
## Step 2: test on data from made-up to real [#step-2-test-on-data-from-made-up-to-real]
We used five test sets, in rising order of realism.
* A synthetic collection. The number of topics per document is a dial, and relevance comes from how the data was built.
* LIMIT, a published stress test that gives a perfect split for free.
* SciFact, real scientific abstracts with human judgements.
* Enron, real company email.
* FollowIR, real news with two sets of human judgements per query.
A technique that works only on made-up data has not been shown to work.
## Step 3: measure in pairs [#step-3-measure-in-pairs]
Every comparison is made question by question. Each gap comes with a 95 percent bootstrap interval. That is the range the true gap most likely sits in. Each also comes with a two-sided paired permutation p-value, the chance a gap this size appears by luck. Where several comparisons run together, we apply Holm step-down correction, which raises the bar. Wins, ties and losses are recorded beside each result.
The reviewer checked the statistics code against textbook definitions over 2,000 random comparisons. The scoring code matches the standard definitions on hand-worked cases. That covers nDCG, recall, MRR, reciprocal rank fusion, alpha-nDCG and subtopic recall. Maximal marginal relevance at lambda 1 reduces exactly to plain relevance ranking, as it should.
## Step 4: write the rules before the results [#step-4-write-the-rules-before-the-results]
Our tests of shaping the index with a goal ran in [five rounds](/patterns/why-not-to-condition-on-the-objective). Rounds 1 to 4 had their rules frozen in writing before any work began. This is called preregistration. The rules covered the hypotheses, the decision rules, the fallback if a baseline would not load, and a stopping rule.
Round 4's rule said, in substance, the following. If business context plus schema metadata did not beat the document sample alone, the claim failed at every level. We would then stop pursuing it. It did not beat it. We stopped.
For round 5, we chose not to preregister, and ran the work to a conclusion. We named the cost at the time. Without frozen rules, a positive result is weaker. Someone could say we tried four designs until one worked.
We kept what discipline we could. Each test's decision rules were fixed in code before any number was read. Every synthetic verdict was computed across several topic-model seeds and reported as a fraction, not from one run. Every setup was reported, including those that beat the method. We record this so readers know which discipline applied to which result.
## Step 5: ask someone to break it [#step-5-ask-someone-to-break-it]
After our first batch of experiments, we ran a reviewer against the material with one instruction: try to reject it. It scored the batch 1.5 out of 5. It reran the experiments rather than only reading the code. It was right on every fatal finding.
The reviewer found that the benefit credited to purpose cards was mostly the benefit of more vectors per document. It also found construction defects in three of the four experiments. Those defects made the headline numbers wrong. The review was the most valuable single output of the work. It made ten findings, five of them fatal. We had found two of the ten ourselves in the same hours, one of them in part. Of the five fatal findings we had missed four, and those mattered most.
| What we reported first | What was wrong | How we caught it |
| ---------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
| Cards beat chunks nine to one on LIMIT. | The chunk control had 4 pieces per document against the cards' 44. | The reviewer reran it with matched counts. nDCG\@10 rose smoothly with pieces per document. |
| Topic dilution followed the theory on synthetic data. | The cards were the exact passages the queries were made from. So the test could not have come out negative. Also, 81 percent of true matches were unlabelled, because filler sentences stated real facts. | The reviewer read the data generator. |
| Cards were a disaster on SciFact (-0.193). | The card builder kept only the top two passages per topic. Cards held 44.6 percent of each document. Chunks held 100 percent. | We measured text coverage after the result looked too extreme. |
| Coverage from varied results was nearly free. | A coding bug gave 2 distractors per task instead of 40. Nothing could be lost, and every relevance score sat at its maximum. | The reviewer. |
| Model readers need more variety than people, and the effect was significant. | It rested on one judge, from the same company as the writing model. | A judge from a different company, at the author's suggestion. |
Two causes run through all five.
**Give the control the same budget.** Three of the five came from a setup quietly weaker than the one it was compared with. It had fewer pieces, less text or no distractors. So the comparison measured the handicap, not the idea.
**Make sure the test can fail.** The first synthetic test could not have produced a negative result, because the cards were the answer key. The first coverage score fired on shared words. Outputs "covered" 4.3 sub-topics from selections that held only 1.8. That is impossible, and spotting it is the check that caught the fault. Ask of any test what result would prove it wrong. If there is none, the test proves nothing.
Some things held up under the same review. The statistics code was sound. Nothing leaked from the test set, since the anchor phrases that build cards never see queries or judgements. And the chunk windows covered 100 percent of the text, so the control never lacked text.
## Step 6: rerun on a second machine [#step-6-rerun-on-a-second-machine]
We first produced the results on an Intel Mac. We then rebuilt everything on an Apple Silicon Mac and compared every regenerated file with the original. This found two defects in our test code. Both are now fixed.
With the fixes, the pure search experiments reproduced byte for byte across the two machines. The dilution, diversity and filter experiments matched to floating-point precision. Results that depend on clustering did not repeat, because the topics came out differently. That includes the round 3 cross-over. Round 4, the sampler test and the null result for the conditioning levels did repeat. The details are in the collapsed note below.
## What would prove the remaining claims wrong [#what-would-prove-the-remaining-claims-wrong]
Each claim that survived names the result that would overturn it.
* *Several vectors beat one on documents about many topics.* Overturned by a real collection with single-topic questions and human judgements, where matched chunks do not beat the pooled vector.
* *Purpose cards help when the card types match the questions.* Overturned by a real collection of that kind, where hand-written or model-designed cards fail to beat matched chunks. The synthetic +0.188 is the only positive so far.
* *A constant saving of about two hundred times.* Overturned by a cost model that prices encoder compute and the different outputs fairly, and finds the ratio below ten.
* *The harvest gives the best coverage for its relevance cost.* Overturned by a collection with natural sub-topic labels, where MMR or a DPP matches its coverage. It would have to do so at equal or lower relevance cost.
* *Harvesting reduces unsupported content.* Overturned by a repeat with a different writing model, where the effect disappears.
## A worked example [#a-worked-example]
A hospital wants to switch its clinical-guideline search to purpose cards. First it runs the three setups from step 1 on its own clinicians' questions. The plain chunks get the same number of pieces as the cards. It writes down in advance what gap would count as a win. It keeps keyword search in every setup. Only then does it read the scores.
## Evidence and limits [#evidence-and-limits]
This method is our own, built while running these tests. The statistics are standard. The reviewer had one instruction, to reject. We ran both machines ourselves, and nobody else has repeated the tests yet.
## Next step [#next-step]
See which predictions held and which did not, in [what we got wrong](/patterns/what-we-got-wrong). Or go back to the [nine context patterns](/patterns).
The second-machine rerun in detail
The sentence-encoder library picked the Apple GPU by default. Under the pinned framework version, that path returned vectors unlike the CPU's. They differed by up to 0.19 (mean cosine 0.993). It shifted one pooled baseline by 0.13. Once it hung for good inside a copy from the GPU back to main memory. The vector cache key did not record the device. So GPU vectors would have silently served every later CPU run. The fix makes the CPU the default and the GPU an explicit choice.
Separately, the hash seed was set after the interpreter had started, where it has no effect. One experiment also sampled its training queries in a set's iteration order. So every process showed the designer a different sample, and each run designed different card types. The fix sorts the set and restarts the entry point with the seed fixed. The committed numbers for that experiment are one random draw. The result held on every draw.
With both fixes, the pure encoder and search experiments reproduced byte for byte across processor types. These were LIMIT, the anchor test and both SciFact runs. They also include the round 1 levels and cross-over, and the two extra reader judges. That is 26 committed files. The dilution, diversity and filter experiments matched to floating-point precision. The largest difference was 4.8e-8. The filter's ROC-AUC moved from 0.933 to 0.936.
Everything after the steps that shrink and cluster the vectors diverged. The topics differed, so the designer prompts differed. That forced 27 new design calls, costing about one US cent in total. The cost exponents moved again. That was the second time they showed up as a side effect of clustering.
One committed pooled baseline on SciFact, under a second encoder, was wrong on the Intel machine. It read 0.550, against 0.6875 here and 0.689 on the model card. So that machine's long-text vectors under that encoder are suspect. The result that depended on it had already been marked unusable.
Explore this topic
* [How we evaluate evidence](/research/method), the general method this page applies.
* [What we got wrong](/patterns/what-we-got-wrong), the full scorecard of predictions.
* The findings and the second-machine rerun were recorded from 2026-08-20 to 2026-08-22.