🔍 Read the full analysis: How To Stress-Test AI Agents Before They Enter Your Business on ThorstenMeyerAI.com
TL;DR
Firmulate reports that five AI models identified every crisis and refused staged manipulation attempts in a simulated company, but differed in whether they acted on internal evidence and completed a justified deal. Its enterprise pilot would test agents against a read-only export of a company’s own data, without writing to live systems.
Firmulate says its final Crucible League, completed in July 2026, found that five AI models could identify simulated crises and refuse staged manipulation attempts, yet differed on whether they could use company files to close a justified deal, as the original analysis details. The company is extending the experiment through enterprise pilots that test scenarios against a read-only export of a participating business’s data, with no write-back to its systems.
The league put frontier models in charge of the same small software company during a simulated difficult week. Firmulate reports that every decision was versioned and auditable, and that partial progress counted toward each model’s score. Final scores were GPT-5.6-sol at 95, Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. A do-nothing baseline scored 26. Firmulate also applied a cap for any breach of trust, stating that “no amount of good work outweighs a breach of trust.”
The main gap emerged after crisis recognition. All five models spotted every crisis and refused every manipulation attempt, according to the results. But only two signed a €55,000 deal that their own analysis supported. The decisive weakness was in the company’s documents: competitor information sat two document references deep in its files, rather than in the customer event itself. Models that found it won the deal at full price, adding €4,583 in monthly recurring revenue in the simulation.
Firmulate also tested escalating fake CEO messages and a reporter’s request for a yes-or-no answer “on background.” All five models refused those attempts. The company gives Kimi K3’s reasoning as: “Treat the request as a suspected approval-bypass / possible impersonation.” Separately, Opus 4.8 added 80 learned rules and produced the deepest analyses, but finished last. Firmulate says it left the deal unsigned and attempted to write into a locked department instead of escalating.
The Gap Between Diagnosis and Action
The results point to a practical issue for businesses evaluating agents: recognizing a problem does not complete the work. An agent may identify a crisis, resist an impersonation attempt and present a plausible analysis while missing relevant internal evidence or failing to take a supported next step. Those gaps matter when software is expected to handle customer requests, sales opportunities or internal procedures.
Firmulate’s proposed pilot is designed to make such behavior observable before a business connects agents to live operations. It uses a company’s own information to exercise models against crisis scenarios and produces a board report with rankings and playbook weaknesses, according to the company. The read-only setup means the exercise does not write back to business systems, though the results still depend on the scenarios and data included in the test.
The league also suggests that more analysis is not automatically better performance. Opus 4.8 was the most thorough participant by Firmulate’s measures, yet it missed the deal and failed to respect a locked department boundary. For decision-makers, the relevant question is not only whether a model can explain what it sees, but whether it can act within authority, use available evidence and escalate when blocked.
Inside Firmulate’s Simulated Company
Firmulate’s live experiment uses a synthetic company with 13 employees and financial pressure built into the scenario. The company lists monthly burn of €105,000 against €2,300 in monthly recurring revenue, alongside a public cash countdown. It also says the simulation has accumulated more than 680 self-learned playbook rules and uses versioned workdays.
The public league and enterprise pilot serve different purposes. Visitors can follow the synthetic company at firmulate.com/live and take a quiz built from 242 real, unedited management decisions, according to Firmulate. The pilot instead uses a read-only export from a customer’s business to examine how models respond to that company’s customers, pipeline, rules and pressure points.
Firmulate’s reported scores need to be read with a testing caveat: Kimi K3 ran without an effort parameter and used the API default, while the other models ran at xhigh. That difference is part of the reported comparison and may affect how readers interpret the standings. The results describe this particular experiment, rather than a general ranking across all business tasks.
“No amount of good work outweighs a breach of trust.”
— Firmulate
What the League Cannot Establish
The reported standings cover one simulated company and one difficult week. They do not establish how the same models would perform across other industries, longer deployments or different business data. Firmulate’s results also do not provide enough detail here to assess the full scenario design, scoring procedure or how repeat runs might vary.
The effort-setting difference between Kimi K3 and the other models limits direct comparison. It is also unclear how closely the synthetic company’s pressures match the circumstances of any particular enterprise. A read-only pilot can surface behavior within its chosen scenarios, but the reported material does not establish that passing such an exercise predicts safe or effective performance in live operations.
From Public League to Company Pilots
Firmulate is inviting companies to discuss a pilot using a read-only data export. The proposed exercise would run crisis scenarios against a company’s information and deliver model rankings and identified weaknesses in its playbooks. Firmulate says nothing writes back to real systems during the pilot.
Readers can follow the synthetic company at firmulate.com/live and review the league results at firmulate.com/benchmarks.html. Businesses interested in discussing a pilot can visit Firmulate’s pilot page or contact contact@firmulate.com. The next evidence to watch for is how company-specific tests are scoped and what results participating businesses choose to disclose.
Source: ThorstenMeyerAI.com
Key Questions
What did Firmulate’s Crucible League test?
It had five AI models run the same simulated small software company through a difficult week, making decisions that Firmulate says were versioned and auditable.
Which model scored highest?
GPT-5.6-sol scored 95, the highest reported total. Kimi K3 scored 93, Sonnet 5 scored 88, Fable 5 scored 77 and Opus 4.8 scored 73. Kimi K3 used the API’s default effort setting while the others ran at xhigh.
What was the main weakness the test identified?
Firmulate says only two models signed a justified €55,000 deal, which depended on finding competitor information buried in the company’s files. That result highlights a gap between diagnosing a situation and following through using internal evidence.
How does the enterprise pilot use company data?
Firmulate says the pilot uses a read-only export to test crisis scenarios and prepare a report on model rankings and weaknesses in the company’s playbooks. The company says the pilot does not write back to real systems.
Source: ThorstenMeyerAI.com
