Comparable Outcome Benchmark Method

Job to be done

Decide whether a set of outcome reports is comparable and private enough to publish as a benchmark.

Required inputs

• Reviewed outcome reports
• Common task and metric
• Method versions
• Organization identifiers kept outside the public dataset
• Date window

Procedure

1. Include only reports using the same task definition, outcome unit, measurement window, and compatible method version.
2. Require at least 10 reviewed reports from at least three independent organizations.
3. Suppress a cohort when one organization provides more than half of the reports.
4. Remove direct and quasi-identifiers and test whether small subgroups can be inferred.
5. Publish sample size, median, interquartile range, date window, inclusion rules, exclusions, and limitations.
6. Version every snapshot and retain corrections or withdrawn cohorts without silently overwriting history.

Outputs

• Cohort definition
• Inclusion/exclusion ledger
• Median and interquartile range
• Concentration check
• Privacy decision
• Versioned snapshot

Publication gate

A result remains private or draft until every applicable gate passes.

• At least 10 reviewed reports
• At least three independent organizations
• No majority contributor
• Same task and metric
• Privacy review passes
• Named reviewer approves the snapshot

Limitations

• A benchmark describes its cohort, not every organization.
• Selection bias and self-reporting may remain.
• A median does not establish why an outcome changed.

Sources and prior-art context

These sources inform the working synthesis. Their inclusion does not imply endorsement.

• NIST AI Risk Management Framework - National Institute of Standards and Technology
• Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile - National Institute of Standards and Technology
• Artificial Intelligence: An Accountability Framework for Federal Agencies and Other Entities - U.S. Government Accountability Office

Change history

• 2026-08-25 - Added multi-organization and concentration gates to the existing sample threshold.