AIThis post was created with the assistance of artificial intelligence (AI).

🔍 Read the full analysis: How Are Perplexity And Intelligence Connected In AI? on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

  • Business-only prices and quantity discounts
  • Tax-exempt purchasing
  • Multiple users, one account, clear invoices
As an affiliate, we earn on qualifying purchases.

TL;DR

OpenAI has published internal measurements of its models’ pretraining perplexity and argues that comparable figures could help developers track progress and discuss shared safety standards. The company links lower reported losses with capabilities in its models, but the data do not show that a particular score guarantees an ability, and its forecasts about future progress remain uncertain.

OpenAI has published internal measurements of its models’ pretraining perplexity, arguing that the metric could help AI developers compare progress and inform shared safety standards. The company relates lower reported losses to capabilities seen in its own models, but says future abilities and the timing of further advances remain uncertain; the figures do not establish that any particular score produces a specific capability.

Perplexity measures how well a model predicts the next token in a held-out dataset. Lower values indicate better predictions on that data. OpenAI says it has tracked the measure over time, including for models beginning with GPT-3.5, and has now made some internal results public. It says the decision involved weighing the competitive cost of disclosing more about its position against the possible value of shared safety measures.

In its account, OpenAI associates reported log-perplexity of about 0.99 for GPT-3.5 and about 0.91 for GPT-4 with stages of model capability. It says later models have reached losses below 0.8, and connects progress across these stages with fluent conversation, productive use of chain-of-thought reasoning, and capabilities in mathematics and cybersecurity. These are the company’s descriptions of its models, not independent evidence that a given score reliably predicts those abilities.

OpenAI also says a model at about 0.76 log-perplexity began producing results on open mathematics problems that the post’s author describes as groundbreaking. The post projects further changes in pretraining research as soon as 2027 and anticipates new capabilities around 0.68–0.7. Those numbers are forecasts, not observed future results. The material provided does not establish what capabilities would appear at those levels.

At a glance
reportWhen: Published in OpenAI’s recent post; futu…
The developmentOpenAI published internal pretraining perplexity measurements and proposed the metric as a possible shared reference for tracking AI progress and informing safety standards.
AI progress · Briefing

Perplexity and Intelligence A shared measure?

OpenAI published internal pretraining measurements and argues they could help developers compare progress and shape shared safety standards. The numbers offer a signal, not a guarantee of future abilities.

≈0.76
Log-perplexity linked to reported open math results
≈0.99
Reported for GPT-3.5
≈0.91
Reported for GPT-4
2027
Possible timing of faster research progress

What the metric measures

Perplexity measures how well a model predicts the next token in a held-out dataset; lower values mean better predictions.

1
Pretrain

A model learns from a large dataset.

2
Measure

Perplexity tracks predictive performance on held-out data.

3
Shape capabilities

Prompting, fine-tuning, and reinforcement learning can elicit or shape what a model does.

OpenAI’s reported progression

The company associates lower log-perplexity with capabilities observed in its models; the pattern does not prove a value causes or guarantees an ability.

Reported levelOpenAI’s account
≈0.99GPT-3.5; fluent conversation
≈0.91GPT-4; productive chain-of-thought reasoning
Below 0.8Later models; mathematics and cybersecurity capabilities
≈0.68–0.7Range where OpenAI anticipates new capabilities

Why a shared measure could matter

Task benchmarks can change or saturate as models improve. Comparable pretraining measurements could give researchers and policymakers a common reference.

Potential value
  • Track predictive performance over time
  • Compare progress if developers use shared methods
  • Inform discussion of common safety standards
What it cannot show alone
  • What a model can reliably do
  • How safely it behaves
  • How much oversight development requires

What remains unresolved

The supplied material does not include the underlying chart or enough detail to independently compare the measurements.

  • No standardized reporting or audit process is specified.
  • No formal industry standard, timetable, or commitments from other developers are announced.
  • The possible 2027 research acceleration and future capabilities are projections, not confirmed outcomes.
WatchMeaningful comparisons depend on shared datasets, reporting methods, independent scrutiny, and evidence connecting the metric to capabilities and risks.
The takeaway
Perplexity may be a useful signal of pretraining progress, but it cannot yet support reliable cross-developer comparisons or settle questions about safety and future abilities.

A Possible Shared Signal for AI Progress

If developers report comparable pretraining measurements, researchers and policymakers could gain a way to follow model progress even as task-specific benchmarks change or become saturated. OpenAI’s argument is that a measure tied to predictive performance may provide a steadier reference across model generations and applications than a shifting collection of downstream tests.

That possibility matters for safety discussions, but a common score would not settle safety questions. Perplexity alone does not show how a model behaves in deployment, whether it can be misused, how reliably it follows safeguards, or how much human oversight it needs. It is one measure of predictive performance, not a complete assessment of capability or risk.

For readers, the practical distinction is between a proposal and an established industry practice. OpenAI has shared some of its own measurements and made a case for wider reporting. The source does not announce an adopted standard or commitments from other developers. Its value for comparison will depend on whether organizations use methods and data that can genuinely be compared.

Products Worth Considering

As an affiliate, we earn on qualifying purchases.

Why OpenAI Tracks Pretraining Loss

OpenAI describes model development as beginning with pretraining on a large dataset, followed by methods such as prompting, fine-tuning, and reinforcement learning to elicit or shape capabilities. It argues that pretraining is useful to monitor because downstream evaluations can change as models improve, making comparisons across time more difficult.

The post uses the history of OpenAI models, including GPT-2, GPT-3, GPT-3.5, and GPT-4, to explain why the company considers the measure worth tracking. It says capabilities associated with particular perplexity levels were not predictable in advance. That history is presented as a reason to monitor the metric, not proof that the same pattern will continue.

OpenAI also raises the possibility that AI systems could contribute to algorithmic research on pretraining. The post’s author says rapid progress in mathematics makes a sudden acceleration in deep-learning research conceivable, perhaps by 2027, while acknowledging differences between the fields. This remains a projection rather than a confirmed development.

“A model at about 0.76 log-perplexity began producing results on open mathematics problems described as groundbreaking.”

— The post’s author

How Comparable Are the Measurements?

The published material described in the source does not provide the underlying chart, full dataset details, or enough information to independently compare OpenAI’s figures with measurements from other developers. It also does not specify a standardized reporting process, audit method, or shared dataset.

It remains unclear how strongly perplexity relates to particular capabilities or safety risks, and whether results across organizations would be comparable if models were evaluated on different data or under different procedures. OpenAI’s examples do not show that a particular value guarantees an ability. The timing and nature of future capabilities, including the proposed possibility of AI-assisted advances in pretraining research, are also unsettled.

The source presents publication as a first step toward a shared scientific basis, but does not announce a formal standard, a timetable, or participation by other developers. Until those details emerge, the figures are best understood as OpenAI’s internal measurements and argument for wider tracking, not an industry-wide yardstick.

Whether Other Developers Join In

The next concrete test is whether other AI developers publish comparable measurements and whether researchers agree on datasets and reporting methods that support fair comparisons. Shared methods and independent scrutiny would help establish whether the metric can serve as a useful cross-company signal rather than a record of one developer’s own models.

OpenAI has not set out a timetable for such a process in the material provided. Further evidence would also be needed to show how changes in perplexity relate to capabilities and risks, and what safety decisions—if any—should follow from particular results. The company’s projected timelines and score ranges should be treated as forecasts unless and until subsequent results confirm them.

Key Questions

What is perplexity in AI?

Perplexity measures how well a model predicts the next token in a held-out dataset. Lower values mean better predictions on that dataset; they do not, by themselves, establish what a model can do.

What measurements did OpenAI publish?

OpenAI reported internal figures including approximately 0.99 for GPT-3.5, approximately 0.91 for GPT-4, later losses below 0.8, and a model at about 0.76 log-perplexity. The source does not provide enough underlying data to independently compare these figures across developers.

Does a lower perplexity score guarantee new capabilities?

No. OpenAI relates lower reported losses to capabilities seen in its models, but the post does not show that a particular score reliably produces a specific ability. The company also says future capabilities are uncertain.

Is perplexity now an industry safety standard?

No formal industry standard is announced in the source. OpenAI argues that the metric could support shared tracking and safety discussions, but adoption by other developers, common reporting methods, and independent scrutiny remain unsettled.

Is an AI research acceleration by 2027 confirmed?

No. The post presents further pretraining advances by around 2027 as a possibility and forecasts capabilities in a roughly 0.68–0.7 range. These are expectations, not confirmed outcomes.

Primary source: OpenAI · via ThorstenMeyerAI.com

You May Also Like

How Much Of HN Is AI?

An analysis of how much content on Hacker News is generated by AI, including current estimates, significance, and ongoing uncertainties.

Google Expands Security Efforts for AI Systems

AIThis post was created with the assistance of artificial intelligence (AI). We’re…

Beating GPT-5.6 Sol On Retrieval With 100X Cheaper Open Models

Open-source models now outperform GPT-5.6 Sol on retrieval benchmarks, achieving similar results at 100 times lower cost, marking a significant shift in AI efficiency.

AI in Healthcare Education: Training the Next Generation of Medical Professionals

Navigating the future of medical training with AI promises personalized learning and immersive simulations, but what ethical dilemmas might arise?