Research findings and what remains open

The useful research question is whether a memory design helps an agent recover the right evidence and finish the task. A headline score is only meaningful when we know what was counted and what was never tested.

What the earlier investigation examined

The guide's author reported testing a verbatim-memory retrieval pipeline on LongMemEval, LoCoMo, and a BEAM ten-million-token configuration. The reported lesson was that retrieval recall, answer quality, and task success must remain separate.

Publicly, MemPalace's benchmark notes explicitly distinguish retrieval recall from question-answering accuracy and disclose tuning on particular failing questions. Project-published documentation, reviewed 18 September 2026. This supports the metric distinction; it does not independently verify the guide author's runs.

What can be retained from the reported results

The earlier report describes three mechanisms worth checking in any evaluation: long source units truncated before encoding, different definitions of recall producing different scores, and a retrieval budget so large that it returns the entire candidate pool.

A toy example makes the last issue clear. If there are eight candidate records and the search returns all eight, the required record is necessarily present. That does not tell us whether the ranking was useful or whether an agent could answer correctly.

Earlier author-reported retrieval results and their limits

These values are preserved as author-reported observations from 13 September 2026, not a new independent replication. A complete publicly linked reproduction package has not been established on this page.

• LongMemEval_S, raw retrieval: 96.6% any-hit recall at five; 91.7% fractional recall on the same ranking.
• LoCoMo, raw retrieval at ten: 60.3%; keyword-boosted retrieval: 88.9%. The earlier report states that returning fifty results included the complete candidate pool and yielded 100% recall.
• BEAM 10M, keyword-boosted retrieval: 24.5% fractional recall and 41.5% any-hit at ten results. At one hundred results, the reported values were 42.3% and 65.9%.

The evidence unit differs between the conversational benchmarks and the reported BEAM configuration. These rows are not answer-accuracy results and are not a vendor ranking. The paid reranker configuration was not run in the author's reported experiment.

Later runs, from September to October 2026, measured answer accuracy and model tokens per question on LoCoMo, LongMemEval, EnronQA and BEAM, with every model call counted. Their write-up is in preparation. The numbers will be added here when it is public.

LoCoMo itself needs a warning. An unreviewed audit found about 6.4 percent of its answer key wrong. Its usual judge model accepted about 63 percent of deliberately wrong answers. That puts a perfect score near 93.6 percent. One vendor already reports 92.5 percent (Penfield Labs audit, 2026). The LoCoMo rows above measure retrieval, not judged answers. The judge problem does not apply to them. The other benchmark warnings are on test memory against the work.

What would make this work independently checkable?

Publish the precise dataset version and split, corpus preparation, source-unit boundaries, encoder configuration, retrieval code, settings, metric definitions, and per-question outputs. Separate reproduced numbers from configurations that were only read in another party's report.

Include a simple baseline and representative failures. For a new memory design, also evaluate answers and completed tasks. Retrieval improvements alone do not establish an enterprise outcome.

What remains open

How well do these choices transfer to operational records, images, signals, and changing permissions? Can a memory resume work without duplicate actions? Does a retained episode improve a new investigation rather than bias it toward an old conclusion? How much cost and human review does each added mechanism create?

Those are questions for future experiments, not results claimed here. New findings should show methods, limitations, and evidence before changing the design guidance.

Two gaps are wider than this guide's own work. Nobody publishes memory accuracy on live traffic. Every published memory score comes from scripted or synthetic conversations. Nobody publishes the cost of keeping memory healthy either. That cost covers extracting, merging, pruning and reviewing memories for each agent over time. The only figure found is a vendor estimate. Nightly merging at 100,000 memories would cost about USD 50 to 100 a month (AWS, 4 September 2026) [vendor]. It is a model of the cost, not a measured fleet. Sources read 14 October 2026.

Next step: see how memory learnings are promoted under control on the learning flywheel. The primary references are in sources and further reading.