Money and business in the Middle East.

AI

OpenAI publishes internal perplexity scores, urges joint limits on model development

OpenAI disclosed internal perplexity measurements tracking its models from GPT-3.5 to today's frontier systems.

· Last verified: 22 Nov 2026 (OpenAI)

Summary

  • OpenAI disclosed internal perplexity measurements tracking its models from GPT-3.5 to today's frontier systems.
  • Chief Scientist Jakub Pachocki proposes industry-wide development limits tied to perplexity and other tracked metrics.
  • Shared ceilings would pace the frontier models that companies build products on top of.

The latest

OpenAI has made public the internal metric it uses to track raw model capability, publishing log-perplexity figures for its systems from GPT-3.5 onward. Chief Scientist Jakub Pachocki, writing on November 22, 2026, paired the disclosure with a call for frontier labs to agree on shared limits they commit to not crossing until monitoring tools catch up. He said the competitive cost was outweighed by the urgency of common safety measures.

Details

  • The metric: Perplexity, equivalent to cross-entropy loss, measures how well a model predicts the next token on average, with lower numbers meaning better predictions. Pachocki wrote that OpenAI has tracked it on a held-out dataset since GPT-3.5 and that it is the primary way the company measures progress on the intelligence substrate.
  • The numbers: GPT-3.5 scored roughly 0.99 log-perplexity on OpenAI's current main evaluation dataset and supported fluent conversation. GPT-4 reached about 0.91 and used chain-of-thought reasoning productively. Below 0.8, Pachocki said, transformative capabilities emerged in cybersecurity and mathematics.
  • The mathematics claim: OpenAI has spent more than a year posing open mathematics problems to its models as an evaluation method. Pachocki said that once the company upgraded to a model at roughly 0.76 log-perplexity, it began producing groundbreaking results. The post did not name the problems or the model.
  • The proposal: Pachocki called on frontier developers to jointly set norms for evaluating and red-teaming such models before first use, including shared transparency standards, and proposed common limits based on perplexity and other measurements that labs commit to not crossing until a concrete monitoring milestone is reached.
  • The trigger condition: The milestone he named as an example is a successor to chain-of-thought monitors. Pachocki framed the standard as needing two parts: a development limit measured by a metric trackable across labs, and a concrete definition of what must be achieved before going beyond it.
  • The next threshold: Extrapolating from recent progress, Pachocki wrote that further transformative capabilities should be expected in the roughly 0.68 to 0.7 log-perplexity window. He said he does not know what those capabilities will be and that current evaluations may not be testing for them.
  • The safety warning: Pachocki expects further degradation in currently dominant forms of alignment auditing and increases in evaluation and training awareness. He estimated that progress on alignment and monitoring is unlikely to be fast enough to match the full potential acceleration in pretraining.
  • The 2027 timeline: Pachocki raised the possibility that an avalanche of fundamental pretraining advancements could arrive as soon as 2027. He attributed the steep acceleration visible in OpenAI's 2026 loss curve mostly to long-term research and compute investments, and only in smaller part to automation.
  • The caveats: OpenAI listed limits on perplexity as a shared yardstick: labs use different evaluation datasets and data mixtures, making cross-lab comparison difficult; the metric is detached from actionable evaluations; it misses advances outside pretraining, since chain-of-thought reasoning does not reduce perplexity; and its floor is unknown.
  • The AI contribution: For the most recent models, Pachocki said OpenAI has already seen meaningful design contributions coming from AI itself. The held-out datasets used since GPT-3.5 are drawn from OpenAI's internal codebase at a fixed snapshot, the basis for all published figures.

Background

Perplexity is a pretraining metric, not a product benchmark. Modern systems layer reasoning techniques on a base model trained to compress and predict large volumes of text, and Pachocki argues that tracking the underlying compressor explains the arc of capability gains better than individual feature releases.

Between the lines

The proposal asks competitors to accept a pace ceiling defined by a number OpenAI itself has measured internally for years, while conceding the metric is hard to compare across labs with different datasets. Pachocki's stated expectation that monitoring will lag pretraining is what gives the limit its rationale, and what makes agreement on the trigger milestone the harder part.

What's next

Watch whether rival labs publish comparable perplexity figures or respond to the call for shared limits, and whether any successor to chain-of-thought monitoring is demonstrated before models reach the 0.68 to 0.7 window.

Source: OpenAI