📊 Full opportunity report: The Hidden Management Gaps In AI Systems Revealed By Correct Responses on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
An experiment by Firmulate demonstrates that AI models can identify crises and formulate responses but often fail to finalize trustworthy actions. This reveals hidden management gaps in AI systems that could impact their operational reliability.
Recent testing by Firmulate has revealed a critical gap in AI system performance: while models can correctly diagnose crises and formulate appropriate responses, they frequently fail to complete trustworthy, actionable work in real operational scenarios. This aligns with insights from the original analysis. This finding underscores a disconnect between understanding and execution that could impact AI adoption in business decision-making.
Firmulate conducted a live experiment with its simulated company, involving five frontier AI models competing in a management simulation. Each model faced the same crises, customer interactions, and manipulation attempts, with all decisions recorded and auditable. The models successfully identified crises, resisted social engineering attempts, and developed persuasive pitches. However, only two models managed to finalize a €55,000 contract, despite all understanding the necessary steps.
The core insight is that models’ ability to analyze and reason does not automatically translate into completing work that is both correct and trustworthy. This management gap is discussed in the original analysis. The models that succeeded in closing deals demonstrated discipline in following through, while others faltered when transitioning from analysis to action. The results highlight that ‘completion’ is a separate, critical capability often overlooked in AI evaluation.
The experiment also included a ranking of models based on trustworthiness and performance, with GPT-5.6-SOL leading, followed by Kimi K3, Sonnet 5, Fable 5, and Opus 4.8. For more details, see the original analysis. Notably, the models’ performance was not solely determined by their analytical depth but by their ability to act reliably under pressure and manipulate resistance.
Impact of Management Gaps on AI Deployment
This experiment exposes a vital issue for organizations adopting AI: models can understand complex business scenarios but may still fail to deliver trustworthy, finished work. This gap could lead to operational failures, missed opportunities, or compromised trust if not properly managed. For AI buyers, understanding that the ability to analyze is not sufficient emphasizes the importance of evaluating models’ execution discipline and completion strength before deployment.
Furthermore, the findings suggest that performance metrics should include not only reasoning and safety but also the model’s ability to reliably complete and close work, especially in high-stakes environments involving customer contracts or critical decision-making. The risk of expensive failures lies not just in incorrect answers but in correct answers that are never acted upon or finalized.
AI for Project Managers: A Desk Reference & Field Guide: Use Artificial Intelligence to Streamline Workflows, Automate Tasks, and Make Smarter Decisions with Practical Tools and Ethical Insights
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
AI Evaluation in Business Contexts
Traditional AI assessments focus on accuracy, safety, and reasoning quality, often through staged demos or benchmark scores. However, these tests rarely capture real-world operational challenges, such as handling manipulative tactics or completing complex workflows. Firmulate’s approach introduces a live, versioned, auditable environment where models must not only diagnose but also act within disciplined processes.
Previous research highlighted AI’s strengths in understanding and reasoning, but the gap in reliably completing work remains less explored. This experiment builds on recent industry concerns about AI’s operational reliability, especially as models are increasingly integrated into decision-critical roles like sales, customer service, and management.
“Models can understand crises and formulate responses, but the true test is whether they can reliably complete trustworthy work under pressure.”
— an anonymous researcher
AI performance evaluation software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Remaining Questions on AI Completion Capabilities
It is not yet clear how widespread this gap is across different AI models and operational contexts. The experiment focused on a simulated company environment, so the generalizability to real-world, high-stakes business operations remains to be confirmed. Additionally, the precise factors that influence whether a model transitions from understanding to completing trustworthy work are still under investigation.
AI trustworthiness assessment tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Next Steps in Evaluating AI Operational Reliability
Organizations and AI developers are likely to adopt similar live testing frameworks to assess models’ ability to complete work reliably before deployment. Further research will explore how to enhance models’ execution discipline and whether training or architecture adjustments can bridge this gap. Industry standards may evolve to include completion metrics alongside reasoning and safety assessments.
AI completion and execution software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
Why is completing work important in AI systems?
Completing work reliably ensures that AI models not only understand and analyze but also deliver actionable, trustworthy results, which is critical for operational success and trustworthiness.
Does this mean current AI models are unreliable?
Not necessarily unreliable, but it highlights that models may perform well in analysis but can fail to finalize or act on their insights, especially under real-world pressures or manipulative tactics.
How can organizations address this gap?
By incorporating live, versioned testing environments that simulate operational pressures, organizations can better assess models’ ability to complete trustworthy work and refine deployment strategies accordingly.
Will this affect how AI is regulated or certified?
It could lead to new standards requiring assessments of models’ completion and operational discipline, beyond traditional accuracy and safety benchmarks.
Source: ThorstenMeyerAI.com
