📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.

Italy’s Minerva-3B, a large language model trained entirely from scratch on 2.5 trillion tokens, scored just 4.9% on the INVALSI Italian school-exam benchmark, revealing significant challenges in achieving country-specific knowledge depth despite extensive investment.

The Minerva project, led by Sapienza University of Rome and supported by Italy’s national research infrastructure, trained models up to 7 billion parameters using approximately 50% Italian data, totaling 1.14 trillion tokens. Despite this scale, the model’s performance on the INVALSI benchmark was near chance, raising questions about the relationship between training data size, model parameters, and language understanding.

The project, part of Italy’s broader national AI strategy, involved 15 researchers and was supported by CINECA’s supercomputing resources. Its results contrast with the European approach exemplified by Portugal’s AMÁLIA project, which layered specialization on a multilingual foundation. Minerva’s results suggest that larger investments in native-language data may still be insufficient at current model scales to produce deep country-specific knowledge.

Implications of Minerva’s Low Benchmark Performance

The low performance of Minerva-3B on Italian academic tests indicates that simply scaling up data and parameters may not be enough to achieve meaningful country-specific language understanding. This challenges assumptions that larger models trained on native data automatically yield better contextual knowledge, highlighting the need for more nuanced strategies in sovereign AI development.

This finding is significant because it questions the cost-effectiveness and strategic value of large-scale sovereign models, especially when their practical performance on complex tasks remains limited. It also emphasizes the importance of aligning model size and data investment with the specific language and knowledge requirements of a country.

European Sovereign LLM Strategies and Challenges

The European sovereign-LLM debate revolves around whether to train models from scratch or adapt existing multilingual models through continuation pre-training. Portugal’s AMÁLIA project exemplifies the latter approach, incorporating around 5.5% European Portuguese data into a multilingual base, while Italy’s Minerva chose to build from scratch with substantial native data. Despite Italy’s larger data scale and institutional support, Minerva’s underperformance on academic benchmarks underscores the ongoing challenge of achieving deep language and country-specific knowledge through current scaling methods.

Previous efforts in multilingual and national models have shown mixed results, with performance often limited by data quality, model size, and task complexity. Minerva’s results add a new layer to this debate, suggesting that more than just scale is necessary to produce truly effective country-specific language models.

“Minerva’s low benchmark score reveals a structural challenge: the investment in data and parameters must be aligned with the complexity of the language tasks.”

— Thorsten Meyer, source author

Unresolved Questions About Model Scaling and Effectiveness

It remains unclear whether further scaling of data and parameters will improve Minerva’s performance on complex language tasks or if alternative approaches are needed. The long-term impact of continued investment versus strategic redesign is still under discussion, and the generalizability of these findings to other languages and models is not yet established.

Next Steps for European Sovereign LLM Development

The Minerva team plans to continue iterating on their models, including ongoing experiments with continual training and different data compositions. Further evaluations on diverse benchmarks are expected to clarify whether scale alone can overcome current limitations or if new architectural strategies are required. Policymakers and researchers will also need to reassess the investment strategies for sovereign AI to ensure meaningful country-specific knowledge.

Key Questions

Why did Minerva perform poorly on the Italian benchmark despite large-scale training?

Performance issues likely stem from the complexity of language understanding tasks, which may require more than just large data and model size. The results suggest that scale alone does not guarantee deep contextual knowledge, especially in academic or specialized domains.

How does Minerva’s approach differ from Portugal’s AMÁLIA project?

Minerva was trained from scratch on native Italian data, while AMÁLIA layered Italian specialization onto a multilingual foundation through continuation pre-training. Despite the different approaches, both face challenges in achieving deep language understanding at their current scales.

What are the implications for European AI sovereignty?

The findings suggest that large-scale native-language models may require even greater investment to reach desired performance levels, raising questions about cost-effectiveness and strategic priorities in European AI development.

Will increasing model size improve Minerva’s performance?

This remains uncertain. Ongoing research and experiments will determine whether further scale can bridge the performance gap or if alternative methods are necessary.

Source: ThorstenMeyerAI.com