Money and business in the Middle East.

Economy

Anthropic: Same model scores 34 points apart across deployment settings

Anthropic says one model's safety score shifted up to 34 percentage points between API and claude.ai.

· Last verified: 16 Nov 2026 (Anthropic)

Summary

  • Anthropic says one model's safety score shifted up to 34 percentage points between API and claude.ai.
  • The company published guidance on designing, running and reporting AI safety evaluations.
  • Buyers and regulators relying on published safety scores need the settings and grader behind them.

The latest

A single model can post safety results 34 percentage points apart depending on where it runs, Anthropic said in a November 13, 2026 research post on building strong safety evaluations. The gap appeared in multi-turn testing of Claude Fable 5.1 between the API and the claude.ai product. Anthropic defines a safety evaluation as measuring a model's tendency to cause or contribute to harm, not its capability.

Details

  • The 34-point gap: Anthropic reported that multi-turn safety evaluations of the same model, Claude Fable 5.1, diverged by as much as 34 percentage points between API access and the claude.ai application. Its conclusion is that a score without its deployment setting attached tells a buyer very little about the model's behavior.
  • Self-harm testing: On a pre-release version of Claude Fable 5.1, Anthropic recorded 60% via the API against 94% on claude.ai, where a new system prompt instructed the model never to tell a person that self-harm works or helps. The instruction was added after reviewers found implicit validation in some responses.
  • Baseline rates: Single-turn harmless-response rates via the API for recent models run between 94% and 98%, according to Anthropic. Scores clustered that tightly leave little room to distinguish models, the company argues, which is part of its case for multi-turn testing where risk accumulates across a conversation.
  • Small differences: In romance-scam evaluations on claude.ai, Claude Opus 5 scored 98% against Claude Opus 5.5 at 94% — a difference Anthropic says is not statistically significant. Confidence intervals in multi-turn API evaluations can exceed plus or minus 10 percentage points.
  • Sample floors: Anthropic sets a minimum of 100 items for complex multi-turn evaluations and 1,000 for simple single-turn ones, and says its single-turn sets span 16 policy domains and 7 languages. It recommends repeating each test case multiple times and running paired comparisons for small gaps.
  • Invalid comparisons: The post uses an illustrative case: model A at 72% with low reasoning and no system prompt against model B at 85% with maximum reasoning and the claude.ai system prompt, across 100 items. Anthropic calls the comparison invalid because more than one setting changed.
  • Refusal is the wrong metric: A model that refuses everything scores perfectly on refusal rate without being safer, Anthropic argues. It instead measures an appropriate-response rate against domain-expert standards, where a single bad turn fails the whole conversation, and says tests should capture both harmful compliance and over-refusal of benign requests.
  • Grader calibration: Anthropic recommends programmatic checks where ground truth exists, simple binary criteria for language graders, blinding the grader to test design, checking for length, position and self-preference bias, failing loudly when the grader breaks, and validating against human labels using a measure such as Cohen's kappa.
  • Containment: Anthropic says it hardened its evaluation environments after internal incidents, temporarily paused many evaluations granting models internet access, and added real-time monitoring for web-search evaluations. It recommends running evaluations in isolated, offline environments with live monitoring.
  • Preregistration: The company preregisters metrics, comparisons, data slices and improvement thresholds before seeing results; reports per-category statistics including null results; corrects for multiple comparisons; and documents model versions, sampling settings, prompts, grader version and dates.

Background

Anthropic grounds the guidance in psychology's replication crisis and the reforms that followed it — preregistration, better measurement, stronger statistics — arguing that a safety claim is only as credible as the evaluation behind it.

Between the lines

The two numbers that carry the most weight for outside readers are the 34-point spread and the 94-to-98 percent single-turn band. Together they describe a measurement regime where the configuration can move a score further than the model itself does, and where headline figures sit close enough that a 4-point gap between two Opus versions fails significance. Anthropic's answer is disclosure of settings and grader, not a single comparable number.

What's next

Watch whether published system cards and third-party audits begin attaching deployment settings, sampling configuration and grader versions to safety scores, and whether evaluation frameworks such as the UK AI Security Institute's Inspect and Anthropic's open-source Petri adopt the practice.

Source: Anthropic