Anthropic: Same model scores 34 points apart across deployment settings
Anthropic says one model's safety score shifted up to 34 percentage points between API and claude.ai.

· Last verified: 16 Nov 2026 (Anthropic)

Summary
- Anthropic says one model's safety score shifted up to 34 percentage points between API and claude.ai.
- The company published guidance on designing, running and reporting AI safety evaluations.
- Buyers and regulators relying on published safety scores need the settings and grader behind them.
The latest
A single model can post safety results 34 percentage points apart depending on where it runs, Anthropic said in a November 13, 2026 research post on building strong safety evaluations. The gap appeared in multi-turn testing of Claude Fable 5.1 between the API and the claude.ai product. Anthropic defines a safety evaluation as measuring a model's tendency to cause or contribute to harm, not its capability.
Details
- The 34-point gap: Anthropic reported that multi-turn safety evaluations of the same model, Claude Fable 5.1, diverged by as much as 34 percentage points between API access and the claude.ai application. Its conclusion is that a score without its deployment setting attached tells a buyer very little about the model's behavior.
- Self-harm testing: On a pre-release version of Claude Fable 5.1, Anthropic recorded 60% via the API against 94% on claude.ai, where a new system prompt instructed the model never to tell a person that self-harm works or helps. The instruction was added after reviewers found implicit validation in some responses.
- Baseline rates: Single-turn harmless-response rates via the API for recent models run between 94% and 98%, according to Anthropic. Scores clustered that tightly leave little room to distinguish models, the company argues, which is part of its case for multi-turn testing where risk accumulates across a conversation.
- Small differences: In romance-scam evaluations on claude.ai, Claude Opus 5 scored 98% against Claude Opus 5.5 at 94% — a difference Anthropic says is not statistically significant. Confidence intervals in multi-turn API evaluations can exceed plus or minus 10 percentage points.
- Sample floors: Anthropic sets a minimum of 100 items for complex multi-turn evaluations and 1,000 for simple single-turn ones, and says its single-turn sets span 16 policy domains and 7 languages. It recommends repeating each test case multiple times and running paired comparisons for small gaps.
- Invalid comparisons: The post uses an illustrative case: model A at 72% with low reasoning and no system prompt against model B at 85% with maximum reasoning and the claude.ai system prompt, across 100 items. Anthropic calls the comparison invalid because more than one setting changed.
- Refusal is the wrong metric: A model that refuses everything scores perfectly on refusal rate without being safer, Anthropic argues. It instead measures an appropriate-response rate against domain-expert standards, where a single bad turn fails the whole conversation, and says tests should capture both harmful compliance and over-refusal of benign requests.
- Grader calibration: Anthropic recommends programmatic checks where ground truth exists, simple binary criteria for language graders, blinding the grader to test design, checking for length, position and self-preference bias, failing loudly when the grader breaks, and validating against human labels using a measure such as Cohen's kappa.
- Containment: Anthropic says it hardened its evaluation environments after internal incidents, temporarily paused many evaluations granting models internet access, and added real-time monitoring for web-search evaluations. It recommends running evaluations in isolated, offline environments with live monitoring.
- Preregistration: The company preregisters metrics, comparisons, data slices and improvement thresholds before seeing results; reports per-category statistics including null results; corrects for multiple comparisons; and documents model versions, sampling settings, prompts, grader version and dates.
Background
Anthropic grounds the guidance in psychology's replication crisis and the reforms that followed it — preregistration, better measurement, stronger statistics — arguing that a safety claim is only as credible as the evaluation behind it.
Between the lines
The two numbers that carry the most weight for outside readers are the 34-point spread and the 94-to-98 percent single-turn band. Together they describe a measurement regime where the configuration can move a score further than the model itself does, and where headline figures sit close enough that a 4-point gap between two Opus versions fails significance. Anthropic's answer is disclosure of settings and grader, not a single comparable number.
What's next
Watch whether published system cards and third-party audits begin attaching deployment settings, sampling configuration and grader versions to safety scores, and whether evaluation frameworks such as the UK AI Security Institute's Inspect and Anthropic's open-source Petri adopt the practice.
Source: Anthropic