Independent AI Tests Need Independent Publication Rights
Matthew Leo · Published October 23, 2026 · AI
OpenAI and Anthropic say outside assessors should be able to test frontier models and reach their own conclusions. A new Reuters report shows why that principle is difficult to turn into a genuinely independent evaluation.
Reuters reported on October 13 that the two companies had sought significant control over what some embedded evaluators could publish, citing people familiar with private discussions. The central dispute was not whether testing should occur. It was how proprietary information would be protected, who could demand redactions and whether a lab could approve a report before publication.
The companies dispute parts of the characterization
Anthropic told Reuters that describing it as seeking significant control did not accurately characterize all of its discussions. It said established evaluators would control their methods, criteria, conclusions and publication decisions. For new evaluator firms, Anthropic said it reserved the right to comment on and approve reports while the relationship is being established.
OpenAI said it was not seeking approval rights over full reports. It described its normal practice as redacting confidential information while allowing evaluators to disclose whether major redactions occurred.
Those responses matter. The underlying negotiations are private, and the reported details rely on unnamed sources. What can be independently checked is the gap between the companies’ public principles and the unresolved operating rules.
Independence has more than one control point
An evaluator can be outside the company and still depend on it in several ways:
- Selection: the lab chooses who receives access.
- Funding: the lab pays for work that no public fund supports.
- Access: the lab decides which models, logs, staff and incidents the evaluator can inspect.
- Methods: security or product constraints can narrow the tests.
- Publication: confidentiality review can remove information needed to understand the result.
Trade-secret protection is legitimate. But a redaction process becomes an editorial control if the tested company alone decides whether a fact is proprietary, reputationally damaging or safety-relevant. The report should therefore disclose the rules, not merely the final score.
What a credible report should disclose
Buyers, regulators and journalists should be able to answer five questions:
- Who selected and paid the evaluator?
- What systems and evidence were withheld?
- Could the evaluator design tests without company approval?
- Who had final authority over conclusions and wording?
- Were material redactions made, and were they disputed?
OpenAI’s published principles say assessors should have deep access, scientific rigour and independence while protecting sensitive information. Anthropic’s description of its Accenture partnership says embedded evaluation is new and that standards for access, reporting and funding do not yet exist. Those admissions are more useful than a generic “independent” label.
Why Canadian buyers should care
Canadian governments, banks, hospitals and large employers increasingly rely on vendors’ safety claims. Procurement teams should request the evaluator’s mandate and conflict disclosures, not just a summary badge. A report that cannot describe serious findings may still be useful internally, but it is weaker public evidence.
Mapletechie previously examined an Anthropic evaluation that affected a real police website and a voluntary AI safety accord whose enforcement remains company-led. Both cases point to the same requirement: scrutiny needs authority, evidence and a publication path.
Read the Reuters investigation, OpenAI’s assessment principles and Anthropic’s embedded-evaluation announcement.
Tags: AI evaluations, AI safety, OpenAI, Anthropic, AI governance