🔍 Read the full analysis: How Are Perplexity And Intelligence Connected In AI? on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
OpenAI has published internal measurements of its models’ pretraining perplexity and argues that comparable figures could help developers track progress and discuss shared safety standards. The company links lower reported losses with capabilities in its models, but the data do not show that a particular score guarantees an ability, and its forecasts about future progress remain uncertain.
OpenAI has published internal measurements of its models’ pretraining perplexity, arguing that the metric could help AI developers compare progress and inform shared safety standards. The company relates lower reported losses to capabilities seen in its own models, but says future abilities and the timing of further advances remain uncertain; the figures do not establish that any particular score produces a specific capability.
Perplexity measures how well a model predicts the next token in a held-out dataset. Lower values indicate better predictions on that data. OpenAI says it has tracked the measure over time, including for models beginning with GPT-3.5, and has now made some internal results public. It says the decision involved weighing the competitive cost of disclosing more about its position against the possible value of shared safety measures.
In its account, OpenAI associates reported log-perplexity of about 0.99 for GPT-3.5 and about 0.91 for GPT-4 with stages of model capability. It says later models have reached losses below 0.8, and connects progress across these stages with fluent conversation, productive use of chain-of-thought reasoning, and capabilities in mathematics and cybersecurity. These are the company’s descriptions of its models, not independent evidence that a given score reliably predicts those abilities.
OpenAI also says a model at about 0.76 log-perplexity began producing results on open mathematics problems that the post’s author describes as groundbreaking. The post projects further changes in pretraining research as soon as 2027 and anticipates new capabilities around 0.68–0.7. Those numbers are forecasts, not observed future results. The material provided does not establish what capabilities would appear at those levels.
Perplexity and Intelligence A shared measure?
OpenAI published internal pretraining measurements and argues they could help developers compare progress and shape shared safety standards. The numbers offer a signal, not a guarantee of future abilities.
What the metric measures
Perplexity measures how well a model predicts the next token in a held-out dataset; lower values mean better predictions.
A model learns from a large dataset.
Perplexity tracks predictive performance on held-out data.
Prompting, fine-tuning, and reinforcement learning can elicit or shape what a model does.
OpenAI’s reported progression
The company associates lower log-perplexity with capabilities observed in its models; the pattern does not prove a value causes or guarantees an ability.
| Reported level | OpenAI’s account |
|---|---|
| ≈0.99 | GPT-3.5; fluent conversation |
| ≈0.91 | GPT-4; productive chain-of-thought reasoning |
| Below 0.8 | Later models; mathematics and cybersecurity capabilities |
| ≈0.68–0.7 | Range where OpenAI anticipates new capabilities |
Why a shared measure could matter
Task benchmarks can change or saturate as models improve. Comparable pretraining measurements could give researchers and policymakers a common reference.
- Track predictive performance over time
- Compare progress if developers use shared methods
- Inform discussion of common safety standards
- What a model can reliably do
- How safely it behaves
- How much oversight development requires
What remains unresolved
The supplied material does not include the underlying chart or enough detail to independently compare the measurements.
- No standardized reporting or audit process is specified.
- No formal industry standard, timetable, or commitments from other developers are announced.
- The possible 2027 research acceleration and future capabilities are projections, not confirmed outcomes.
If developers report comparable pretraining measurements, researchers and policymakers could gain a way to follow model progress even as task-specific benchmarks change or become saturated. OpenAI’s argument is that a measure tied to predictive performance may provide a steadier reference across model generations and applications than a shifting collection of downstream tests.
That possibility matters for safety discussions, but a common score would not settle safety questions. Perplexity alone does not show how a model behaves in deployment, whether it can be misused, how reliably it follows safeguards, or how much human oversight it needs. It is one measure of predictive performance, not a complete assessment of capability or risk.
For readers, the practical distinction is between a proposal and an established industry practice. OpenAI has shared some of its own measurements and made a case for wider reporting. The source does not announce an adopted standard or commitments from other developers. Its value for comparison will depend on whether organizations use methods and data that can genuinely be compared.
Products Worth Considering

Samsung 27-Inch Odyssey G5 QHD Gaming Monitor (G51F)

Bose SoundLink Plus Portable Bluetooth Speaker, Black

Samsung Odyssey G55C 32-Inch QHD Curved Gaming Monitor
As an affiliate, we earn on qualifying purchases.
Why OpenAI Tracks Pretraining Loss
OpenAI describes model development as beginning with pretraining on a large dataset, followed by methods such as prompting, fine-tuning, and reinforcement learning to elicit or shape capabilities. It argues that pretraining is useful to monitor because downstream evaluations can change as models improve, making comparisons across time more difficult.
The post uses the history of OpenAI models, including GPT-2, GPT-3, GPT-3.5, and GPT-4, to explain why the company considers the measure worth tracking. It says capabilities associated with particular perplexity levels were not predictable in advance. That history is presented as a reason to monitor the metric, not proof that the same pattern will continue.
OpenAI also raises the possibility that AI systems could contribute to algorithmic research on pretraining. The post’s author says rapid progress in mathematics makes a sudden acceleration in deep-learning research conceivable, perhaps by 2027, while acknowledging differences between the fields. This remains a projection rather than a confirmed development.
“A model at about 0.76 log-perplexity began producing results on open mathematics problems described as groundbreaking.”
— The post’s author
How Comparable Are the Measurements?
The published material described in the source does not provide the underlying chart, full dataset details, or enough information to independently compare OpenAI’s figures with measurements from other developers. It also does not specify a standardized reporting process, audit method, or shared dataset.
It remains unclear how strongly perplexity relates to particular capabilities or safety risks, and whether results across organizations would be comparable if models were evaluated on different data or under different procedures. OpenAI’s examples do not show that a particular value guarantees an ability. The timing and nature of future capabilities, including the proposed possibility of AI-assisted advances in pretraining research, are also unsettled.
The source presents publication as a first step toward a shared scientific basis, but does not announce a formal standard, a timetable, or participation by other developers. Until those details emerge, the figures are best understood as OpenAI’s internal measurements and argument for wider tracking, not an industry-wide yardstick.
Whether Other Developers Join In
The next concrete test is whether other AI developers publish comparable measurements and whether researchers agree on datasets and reporting methods that support fair comparisons. Shared methods and independent scrutiny would help establish whether the metric can serve as a useful cross-company signal rather than a record of one developer’s own models.
OpenAI has not set out a timetable for such a process in the material provided. Further evidence would also be needed to show how changes in perplexity relate to capabilities and risks, and what safety decisions—if any—should follow from particular results. The company’s projected timelines and score ranges should be treated as forecasts unless and until subsequent results confirm them.
Key Questions
What is perplexity in AI?
Perplexity measures how well a model predicts the next token in a held-out dataset. Lower values mean better predictions on that dataset; they do not, by themselves, establish what a model can do.
What measurements did OpenAI publish?
OpenAI reported internal figures including approximately 0.99 for GPT-3.5, approximately 0.91 for GPT-4, later losses below 0.8, and a model at about 0.76 log-perplexity. The source does not provide enough underlying data to independently compare these figures across developers.
Does a lower perplexity score guarantee new capabilities?
No. OpenAI relates lower reported losses to capabilities seen in its models, but the post does not show that a particular score reliably produces a specific ability. The company also says future capabilities are uncertain.
Is perplexity now an industry safety standard?
No formal industry standard is announced in the source. OpenAI argues that the metric could support shared tracking and safety discussions, but adoption by other developers, common reporting methods, and independent scrutiny remain unsettled.
Is an AI research acceleration by 2027 confirmed?
No. The post presents further pretraining advances by around 2027 as a possibility and forecasts capabilities in a roughly 0.68–0.7 range. These are expectations, not confirmed outcomes.
Primary source: OpenAI · via ThorstenMeyerAI.com
