# What we got wrong
The scorecard for every prediction in our context experiments. Three held, one held only in a narrower form, and seven were contradicted or not supported. One useful result came by accident.
Author: Murali Sid (https://linkedin.com/in/muralisid)
Source: https://www.agenticarchitectureskills.com/patterns/what-we-got-wrong
Published: 2026-08-24
Updated: 2026-10-15
Licence: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/
Cite as: Murali Sid. What we got wrong. Agentic Architecture Skills. https://www.agenticarchitectureskills.com/patterns/what-we-got-wrong
---
Research write-ups usually show the ideas that worked. A page of wins alone gives you no way to judge how careful the wins were. So this page shows all eleven of our predictions, including the ones that failed. Three held. One held only in a narrower form. Seven were contradicted or not supported. One useful result turned up by accident. The idea that sounded most impressive is the one that did not survive.
These predictions came from our own tests on how to give an agent the right material. That material is what agents draw on through [knowledge and memory](/memory). The patterns that survived are on the [context patterns](/patterns) page. They apply to the work of the [agent data engineering layer (R14)](/layers/r14-agent-data-engineering) along the [data-to-memory pipeline](/architecture/data-to-memory).
## The scorecard [#the-scorecard]
A purpose card is one searchable piece per topic in a document. Harvesting means keeping the most relevant results and filling the rest with the least similar ones.
| What we predicted | What happened | Verdict |
| -------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- |
| Several vectors per document beat one blended vector when documents cover many topics. | Held clearly. The blended vector falls from 0.815 to 0.294 as topics grow. Several vectors stay above 0.63. | **Confirmed** |
| The gain comes from shaping each vector around a purpose, not just from having more of them. | Failed. A three-word window that ignores meaning recovers most of the gain. Piece count explains it. | **Contradicted** |
| Purpose cards beat a fair like-for-like control on real text. | Failed. On scientific abstracts the cards scored 0.635 against plain chunks' 0.667. They lost 70 comparisons to 36. | **Contradicted** |
| Keyword search can be dropped once the vectors are good. | Failed, and not by a little. On queries naming a specific thing, keyword search scored 0.997 against 0.314. | **Contradicted** |
| A cheap filter separates on-topic from off-topic well enough to be worth building. | Held, at 0.933 on the usual measure. | **Confirmed** |
| That filter is where the large cost saving comes from. | Failed. Keeping 95 percent of relevant material discards only about a third of the documents. That saves one and a bit times. | **Contradicted** |
| Paying per topic instead of per document is a saving that grows with collection size. | Half failed. The growth was not real. It came from our own topics getting coarser. A large constant saving, about 200 times, survives. | **Partly contradicted** |
| Harvesting varied results buys topic coverage more cheaply than the standard method. | Held. About twice as cheaply, with fewer answerless documents let in. | **Confirmed** |
| How much variety to return depends on whether a model or a person reads the results. | Failed. The effect vanished under a judge from a different company. | **Not supported** |
| Designing the card types around your research goal beats deriving them from the documents. | Failed repeatedly, across five rounds. A cross-over test swapped the goals both ways and found no reliable gain. | **Not supported** |
| Letting a model design the card types beats hand-written ones. | No difference. Both stayed below the plain control on real text. | **Not supported** |
## The one that mattered most [#the-one-that-mattered-most]
The most novel claim failed. The way it failed is the useful lesson.
The idea was that the right amount of variety in search results depends on who reads them. A model writing a summary would want more variety, because models make use of odd clues. A busy person would want a tighter list, because extra material is noise to them.
It measured positive at first. Then it measured smaller under a more capable judge from the same company. Then it disappeared under a judge from a different company. A judge is a model that grades another model's work. An effect that shrinks as the grader moves away from the writer's company is a model marking its own homework. It is not a property of the world.
It is not disproved. All three estimates pointed the same way, and sixty tasks cannot settle something this small. But no evidence can be cited for it, and it is not in the patterns. See [the measurement](/patterns/diversity-and-consumer).
## The accident [#the-accident]
Our most durable result was not one we set out to test. We were measuring whether varied results help a model write better summaries. Along the way we also counted statements in those summaries that the retrieved documents did not support. Harvesting varied results reduced that unsupported content. Unlike the finding above, this held under judges from every company we tried.
Nobody predicted it. It is now the most solid single result of this work. It points at grounding, meaning answers backed by their sources, rather than at relevance. That is not where we were looking.
## How we caught our own mistakes [#how-we-caught-our-own-mistakes]
We publish the failures so you do not have to take our word for the wins.
Five headline results reversed at least once. Every reversal traced to one of two causes. Either a control was not matched, so the comparison measured a handicap rather than an idea. Or a measurement could not have come out negative. Those two checks are the part worth copying. They are written up in [how to test a context design](/patterns/how-to-test-a-context-design).
We also pointed an independent reviewer at our first batch of experiments, with instructions to reject it. It scored the batch 1.5 out of 5. It made ten findings, five of them fatal. We had found two of the ten ourselves, one of them in part. Of the five fatal findings we had found one. We had missed the other four, and those mattered most.
## What this means for your own system [#what-this-means-for-your-own-system]
Take a utility building search for its field engineers. Keep keyword search, because asset IDs and work order numbers are exact strings. Split long documents into several pieces. Start with plain chunks, and add purpose cards only if they win on the engineers' own questions. Do not tune variety to who reads the results. These steps follow from the scorecard. We did not test utility data.
## Evidence and limits [#evidence-and-limits]
These pages summarise experiments we ran ourselves, and our corrections. Our working records are not offered as independent evidence. Use the stated methods and limits to decide what to reproduce on your own tasks.
The encoders were small. The collections ran to twenty thousand documents. Everything was in English, and no production data was used. Prices are as of 2026-08-19, and local compute is priced at zero. Whether a larger instruction-tuned encoder strengthens the query-time result is still to be tested. The obvious next test is a real collection with single-topic questions and human judgements. It would test the one positive finding that still rests on synthetic data.
## Next step [#next-step]
See the checks that caught these mistakes, in [how to test a context design](/patterns/how-to-test-a-context-design). Or see [what changed our advice](/research/what-changed).
Explore this topic
* [The decision on purpose cards and chunks](/decisions#cd-25) records the decision this evidence supports.
* The experiment behind each verdict is linked from the [context patterns](/patterns). The experiments ran in August 2026.