METR’s Anthropic Access Puts AI Evaluation Terms in Public

Anthropic said on October 15 that six employees of the nonprofit METR now have company hardware, office space, and access to internal systems roughly comparable to its own risk assessment teams. The arrangement gives outside researchers a closer view of how the Claude developer evaluates risk. Its most important provision is a publication right: Anthropic says it cannot veto METR’s findings, although it can require defined categories of sensitive information to be removed.

The announcement makes a previously abstract debate concrete. Independent evaluation depends on what researchers can inspect, what they can say afterward, and whether readers can discover what was withheld. Access alone cannot settle those questions. A report can be technically competent and still give an incomplete picture if its evidence was selected by the organization under examination.

What the agreement actually offers

Anthropic’s announcement excludes customer data, legally privileged material, and model architecture information from ordinary access. Architecture details may be shared when relevant, subject to internal approval. METR can publicly describe and dispute redactions without revealing the protected content. These are the company’s published descriptions of the arrangement; the full agreement is not included in the announcement.

Anthropic also describes three access tiers. The first requires publication approval. The second permits publication subject to category-based redactions that evaluators can publicly challenge. The third adds the deeper access described for METR. Consequently, an “independent evaluator” label does not tell a reader which rights apply. The terms of each engagement matter.

Reuters reported the arrangement on October 15 and included METR’s commitment to disclose its scope, access, and limitations. That independently corroborates the announcement and the evaluator’s stated position. It does not independently verify the completeness of Anthropic’s internal evidence or the effectiveness of its safeguards.

METR draws a narrower boundary

In an October 14 proposal, METR describes assessments covering agent behavior, monitoring effectiveness, and the acceleration of AI research. It seeks private staff interviews, access to relevant internal records, and timely public reports. But it explicitly says access remains at developers’ discretion and that these exercises cannot provide confident assurance that development is safe.

The distinction between Anthropic’s “employee-like” wording and METR’s caution should remain visible. A researcher may receive a laptop and broad cooperation while lacking the authority to demand a particular dataset. A denied request can be reasonable, especially where personal information or security details are involved. It can also prevent a conclusion. The reader needs to know which happened.

METR’s proposed work excludes several subjects, including model-weight security, sandbox robustness, and organizational compliance. It does not make deployment approval decisions. An assessment of selected loss-of-control questions therefore cannot stand in for a comprehensive security audit. Treating it that way would attach a stronger conclusion to the report than its authors claim.

A redaction statement is part of the evidence

METR’s separate statement of its working practices says readers should learn the scope, access restrictions, redactions, conflicts, and pressures affecting an assessment. It describes retaining a unilateral redaction summary and says it has declined frontier-lab funding while accepting model access that it could afford to purchase. The nonprofit also acknowledges continuing dependence on companies’ willingness to cooperate.

That disclosure mechanism is valuable because confidentiality and accountability need not be mutually exclusive. An evaluator can say that missing evidence materially limits a conclusion without publishing a customer record or an exploitable vulnerability. The public may not need every underlying byte. It does need an honest account of whether the unavailable bytes could change the answer.

Consider a hypothetical report concluding that a monitoring system detected most observed unsafe actions. Its meaning changes if the evaluator examined all production agent traffic, a selected sample, or only incidents already flagged by the monitor. The same percentage can describe very different coverage. An explicit account of sampling and missing records prevents a reassuring number from carrying more weight than it deserves.

Persistent records make scrutiny possible

For agent operators, the practical implication is architectural. Investigators need records connecting an action to the model version, permissions, task, tool response, and approval state that produced it. A polished incident summary cannot substitute for the underlying sequence when the question is whether a safeguard actually intervened.

Those records also need a history. If a permission changed halfway through a run, the investigator must recover the policy active at that moment. If a transcript was compacted, the retained summary should be distinguishable from the original exchange. If an incident label was revised, the previous classification should remain attributable. Otherwise later interpretation can quietly replace the evidence it was meant to explain.

This is editorial analysis, not a claim that the new agreement implements those controls. The published terms establish access and disclosure commitments; they do not demonstrate a complete provenance system. The distinction matters for any organization evaluating agents, including smaller teams whose evidence may be scattered across chat histories, application logs, and tool providers.

Retention should remain proportionate. An independent reviewer does not need unrestricted copies of every conversation, and preserving evidence does not justify indefinite storage of personal data. A useful design can separate operational memory from protected audit records, restrict access, record omissions, and define expiration. Its test is whether another qualified person can reconstruct a consequential decision within the authorized scope.

The first reports will test the promise

The arrangement deserves attention because it exposes actual publication terms at a time when outside evaluation is increasingly presented as evidence of responsible development. It also creates observable questions: will reports identify denied access, explain redactions, document unresolved disagreements, and appear promptly enough to inform decisions?

A critical report would not by itself prove the arrangement failed. Suppressed or undisclosed limitations would be more troubling. Conversely, a favorable report should earn confidence through its methods and evidence rather than the evaluator’s presence inside the company. The useful outcome is a public record that lets readers judge what was examined, what remains unknown, and how strongly each conclusion follows.

Sources

• Anthropic: An update on independent evaluators
• METR: Proposal for public evidence about loss-of-control risks
• METR: Our approach to working with AI companies
• Reuters: Anthropic grants METR employee-level access