Updated 2026-10-20
from the news desk
How should AI labs be independently evaluated? Anthropic tries embedded third-party review
Anthropic said on 15 October 2026 that six METR staff now have company hardware, office space and access roughly comparable to Anthropic’s risk teams, with publication rights subject to predefined redactions. Anthropic’s 19 October roadmap…
Anthropic’s 15 October notice suggests one answer to how AI labs should be independently evaluated: give a small outside team access roughly comparable to internal risk staff, and let that team publish without company veto except for predefined redactions. Anthropic’s 19 October roadmap update also says embedded third-party evaluators are now a formal policy goal, which makes this more than a general call for outside scrutiny.
What Anthropic said on 15 October
Anthropic wrote on 15 October: “Last month, we committed to stronger independent, third-party assessment of our models and safety practices.” It then said: “Today, we’re announcing that we’ve started to give employee-like access to a team from METR, a nonprofit that evaluates frontier AI models.” The same notice states that METR will publish the results of its risk assessments, and that Anthropic has worked with METR over the past year on several reviews of Anthropic Risk Reports.
Anthropic’s notice says six METR employees were onboarded with company hardware, office space, and access to internal tools and systems roughly on par with Anthropic’s own risk assessment teams. Anthropic adds that, outside a small number of defined categories, its employees are encouraged to share information with that METR team as they would with a fellow employee working on alignment risk.
On publication rights, Anthropic’s notice says METR can publish analysis of Anthropic’s risk levels, incidents, or practices, and that Anthropic cannot veto publication. The same notice also states that Anthropic can require redactions if material falls into predefined categories such as reproducing confidential intellectual property, and that Anthropic ultimately decides whether information falls into those categories. Anthropic adds that METR reserves the right to make certain statements about the level of access and information it received.
Anthropic’s notice also narrows the arrangement. It says the engagement is limited to a small team operating under enforceable agreements, and it states that other METR employees are siloed from that team’s work, including researchers who work with Anthropic on other projects. Anthropic describes this as a security-bounded model rather than open access for an outside organization.
Why Anthropic says this model is needed
Anthropic explains the rationale in risk terms. Its notice says: “As models have become more powerful, they’ve required stronger safeguards and more challenging risk assessments.” The same document adds: “We want to be publicly accountable for the decisions we’re making.” Anthropic also says it cannot publicly share all information relevant to safety work because some of it could aid malicious actors or reveal commercially sensitive details.
Anthropic’s notice states that outside evaluators therefore need deep access, plus the freedom to publish observations and assessments. On limits to public disclosure, Anthropic writes: “Some of it could be useful to malicious actors, or is commercially sensitive, like compute or training information.” The notice then says Anthropic is trying to close the gap between what a frontier AI company knows about its own systems and what outside evaluators can know.
Anthropic also sets out what it says evaluators need: expertise, independence, enough access to assess risk meaningfully, and freedom to publish. The notice compares that logic with bank supervision, and states: “If a bank’s risk-taking could threaten the public, third parties try to see that in advance.” One reading is that Anthropic is treating independent evaluation less as a one-off audit and more as resident examination inside a high-risk institution.
A second reading cuts the other way. Anthropic’s own description shows that independence here is partial rather than absolute: the company defines excluded categories, decides whether some redactions are justified, and limits the arrangement to a small embedded team. In that sense, the case for the model rests on whether deep access plus publication rights outweigh company control over some boundaries.
What the 19 October policy update changes
Anthropic’s Responsible Scaling Policy updates page says that, on 19 October 2026, it replaced an earlier policy goal with two new ones. One, the updates page states, is a policy proposal on embedded third-party evaluators due 15 December 2026. The other, the same page states, is a policy framework for targeting the pace of AI development due 15 January 2027.
That matters because Anthropic’s 15 October notice described three evaluator tiers. Anthropic says early-stage partners may get narrower access and more limited publication rights, while the top tier matches the METR design. As Anthropic puts it: “The third tier is the arrangement we’ve described with METR, with employee-like access and publication rights similar to those of the second tier.” Anthropic also says it hopes over time to have several evaluators at that third tier, each publishing its own analysis of Anthropic’s risk posture and key decisions.
For readers tracking what AI labs reveal in system cards, or asking whether AI welfare can affect safety, the practical point is simple. Anthropic’s two October documents describe a governance design in which outside reviewers are embedded inside the lab, but are still expected to publish under their own names. Anthropic’s updates page does not yet say what its forthcoming policy proposal will contain, so the broader model remains unsettled. Readers looking specifically at Anthropic can compare this governance move with what Anthropic publishes about Claude.
Frequently asked questions
What access does METR have inside Anthropic?
Anthropic’s 15 October notice says six METR employees have company hardware, office space, and access to internal tools and systems roughly on par with Anthropic’s own risk assessment teams, subject to defined exclusions.
Can Anthropic stop METR from publishing its findings?
Anthropic’s notice says Anthropic cannot veto publication. The same notice says Anthropic can require redactions for predefined categories, such as reproducing confidential intellectual property, and that Anthropic ultimately decides whether information falls within those categories.
Did Anthropic make embedded evaluators a formal policy goal?
Anthropic’s Responsible Scaling Policy updates page says that on 19 October 2026 it added a policy proposal on embedded third-party evaluators, due 15 December 2026.
Are Anthropic’s other evaluator partnerships at the same level as METR?
Anthropic’s 15 October notice says its other evaluator partnerships are at earlier stages, with more limited information access and more limited publication rights. Anthropic says it hopes to fully embed more independent evaluators over time.
caveats: Nearly all material here, including the access terms, redaction rules, team size, and policy-goal timing, comes from Anthropic’s own publications. No METR document in the record confirms the arrangement from METR’s side.
Sources
- Anthropic — An update on independent evaluators — 2026-10-15
- Anthropic — Frontier Safety Roadmap Updates — 2026-10-19