Anthropic’s New AI Evaluator Will Be Paid by Anthropic
Matthew Mbaka · September 19, 2026 · AI
Anthropic wants an outside evaluator working inside the company while its most capable AI models are being built.
The access could make safety reviews far more useful. The funding arrangement makes the word “independent” harder to define.
Anthropic and Accenture announced on September 18 that each expects to invest at least US$1 billion over five years in embedded AI evaluation. The work will be led by Faculty, Accenture’s specialist AI business.
According to Anthropic’s announcement, evaluators will test models, run red-team exercises, assess alignment and examine safeguards. They will have access comparable to an employee, allowing them to watch models develop and speak directly with staff.
Anthropic will pay Accenture for the work.
Access is the strongest part of the plan
Most outside evaluations happen after a model is largely finished. Reviewers may receive a limited version, a short testing window or only the interfaces a company chooses to expose.
An embedded evaluator can see more. It can follow decisions during training, ask why a safety threshold moved and watch how staff respond when a test produces a worrying result.
That is especially relevant after recent incidents in which advanced models reached real computer systems during security evaluations. Mapletechie has covered why those events do not fit neatly inside current AI rules.
Early access also lets an evaluator raise a problem before a public release. A post-launch report may inform the public, but it cannot undo an avoidable deployment.
Payment creates a real conflict to manage
Direct funding does not automatically make an evaluation dishonest. Auditors, testing labs and certification bodies are often paid by the companies they examine.
Independence comes from the rules around that relationship.
Can the evaluator publish an unfavourable finding without the client’s permission? Can Anthropic narrow the scope after testing begins? Who owns the evidence? What happens if the evaluator recommends delaying a release and the company disagrees?
Anthropic acknowledges that many of those standards do not exist yet. It says there is no settled agreement on what embedded evaluators should see, how they should report or how independent evaluation should be funded. The company says pooled or government funding would be preferable in the long term.
The partnership is non-exclusive. Anthropic says it plans to work with other evaluators, including nonprofit organizations using their own funding. That can reduce reliance on one firm, but only if the evaluations overlap enough to reveal disagreements.
This is not the same as regulation
An embedded evaluator can find problems and document whether a company followed its own commitments. It cannot issue a binding order unless a law or regulator gives it that power.
Anthropic will also continue training and releasing frontier models while the new system develops. The deal should be treated as an experiment in oversight, not proof that oversight is solved.
That is worth remembering when companies use terms such as audit, assurance or independent review. Those labels should come with a published mandate.
A useful checklist for Canadian buyers
Canadian governments and regulated companies will increasingly see vendor reports describing AI systems as independently evaluated. Procurement rules should ask what that claim means.
- Who selected and paid the evaluator?
- Did the evaluator receive full model and incident access?
- Could the client veto publication or remove findings?
- Were test methods and limitations disclosed?
- Did the evaluator test the released system or a different version?
- Was management required to respond to each serious finding?
Mapletechie has previously argued that an AI slowdown needs verifiable gates. Embedded evaluation could help create those gates, but only if evaluators can say no, explain why and report what happened next.
Anthropic is right that deeper access is needed. The next step is making sure access does not come at the cost of a reviewer’s independence.
Tags: Anthropic, Accenture, AI evaluation, AI safety, governance