How we evaluate evidence

Start with a decision

State what would change if the proposed approach worked. Define the task and the outcome before selecting a benchmark. Compare against the simplest credible alternative that could meet the requirement.

Keep the comparison fair

Match data access, capacity, task scope, and evaluation conditions where appropriate. Separate tuning cases from evaluation cases. Include failures, no-improvement results, and the cost of preparing data and reviewing output.

Distinguish sources of evidence

An external paper, a vendor report, practitioner experience, and an author-run experiment offer different kinds of evidence. Label them. A vendor's documentation can establish a described capability without independently proving its business benefit.

Original measurements in this guide are attributed to the author and described in their public reports. They should not be treated as independent replication. External citations point to the underlying paper, documentation, or standards source; internal working files are not reader-facing evidence.

Report enough to understand the result

State the question, data, method, baseline, measures, result, and limits. Name any missing artifacts or reproducibility constraints honestly. Do not replace an unavailable primary source with an internal note that repeats its claim.

Change guidance carefully

Connect a finding to the affected design choice. Update the recommendation, its scope, review date, and relevant skills. Keep the historical result intact and explain why the interpretation changed.

Review current claims

A content edit date is not a fact-check date. Product capabilities, availability, prices, legal requirements, and benchmark comparisons need an appropriate review interval. If a claim cannot be checked, narrow it or make the uncertainty explicit.

See NIST's AI RMF Playbook for an external risk-management resource (accessed 17 September 2026). The research method described here is the guide's editorial approach.

Next step: see the method applied on experiments and results, or return to Start here.