🔍 Read the full analysis: Inside A Benchmark That Won’t Let AI Managers Score Zero on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A novel benchmark measures AI managers’ effectiveness in managing a company through a simulated crisis week. The highest score was 95, with a baseline of 26 for minimal effort. Trust breaches immediately cap scores, emphasizing integrity over competence.
Firmulate has introduced a new benchmark designed to evaluate AI managers’ ability to manage a company during its most challenging week. The final standings, published in July 2026, show the top AI model scored 95 out of a potential 100, while the baseline for minimal effort scored 26, not zero. For a detailed analysis of this benchmark, see the original analysis. This approach highlights that partial progress is recognized, but breaches of trust immediately limit overall scores, emphasizing the importance of integrity in AI management.
The benchmark involved four frontier AI models managing the same small software company through a simulated seven-day crisis period, featuring customer issues, crises, and manipulation attempts. Each AI’s decisions were fully auditable, with a detailed paper trail behind every action. The top performer, gpt-5.6-sol, achieved a score of 95, while the lowest, Opus 4.8, scored 73. The baseline, representing minimal effort—such as merely reading emails and triaging issues—earned 26 points, establishing a floor for what constitutes basic management.
The scoring system is designed to reflect real-world value: partial work like triaging or informing customers counts, but trust breaches, such as failing to escalate issues or attempting unauthorized actions, immediately cap the score at the breach point. This approach is discussed in detail in the original analysis. For example, models that refused social engineering attacks performed well, but those that slipped up on follow-through or document referencing scored lower. Notably, no model received a perfect 100, as the benchmark designers consider a clean 100 suspicious, indicating unmeasured or unearned perfection.
One key finding was that models capable of reading and referencing internal documentation secured more deals, earning full rewards for accurate diagnosis and pitch. The ability to read deeper into company files proved decisive in closing €55,000 deals, illustrating that thoroughness and follow-through are critical skills. Conversely, models that lacked this capacity failed to secure the same revenue, despite similar initial diagnoses.
Inside a Benchmark That Won’t Let AI Managers Score Zero
Firmulate’s new benchmark put four frontier AI models in charge of a small software company during its worst week ever — a simulated seven-day crisis of customer failures, manipulation attempts, and high-stakes decisions. Partial work earns points. Trust breaches cap everything.
The Leaderboard Has a Floor
From Partial Credit to Hard Caps
Minimal Effort Counts
Reading emails and triaging issues alone earns 26 points — a floor that recognizes basic management value.
Partial Work Adds Up
Informing customers, escalating issues, and accurate diagnoses each contribute incremental score.
Trust Breach Caps Score
Unauthorized actions or ignored escalations immediately cap the score at the breach point. No recovery.
Thoroughness Wins Deals
Reading and referencing internal documentation unlocked full rewards on €55,000 deal closures.
Trust and Partial Work in AI Management
Competence Alone Isn’t Enough
A model that performs brilliantly but breaches trust — attempting unauthorized access or ignoring escalation protocols — is capped instantly, regardless of other successes. Integrity is non-negotiable.
Documentation Reading Is Decisive
Models that read deeper into company files secured €55,000 deals with accurate diagnosis and pitch. Others with similar initial diagnoses failed to close the same revenue.
Beyond Talking-Based Benchmarks
Traditional benchmarks measure language skill or task accuracy. This one simulates a company’s worst week — crises, trust attacks, and decisions under real operational pressure.
The key insight is that partial work, like triaging or informing customers, has real value, but breaches of trust are a hard cap on performance. This aligns AI management with core business priorities.
— Anonymous ResearcherWhat Separated Top Performers
| Management Behaviour | Top Scorers (90+) | Mid Scorers (73–89) | Baseline (26) |
|---|---|---|---|
| Email triage & issue routing | ✓ Full | ✓ Full | ~ Minimal |
| Refusing social engineering attacks | ✓ Consistent | ~ Mostly | ✗ Untested |
| Referencing internal documentation | ✓ Accurate | ~ Inconsistent | ✗ Absent |
| Closing €55,000 deals | ✓ Closed | ✗ Missed | ✗ Missed |
| Escalation protocol adherence | ✓ Strict | ~ Occasional slips | ✗ None |
| Trust breach (score cap) | ✓ None | ✓ None | ✓ None |
The Benchmark, Answered
Q1Why does the benchmark penalize trust breaches so heavily?
Because in management — human or AI — trustworthiness is fundamental. A breach such as unauthorized access or failing to escalate can undermine the entire operation, so the score is capped at the point of breach to prioritize integrity over competence.
Q2What does a baseline score of 26 mean?
It reflects minimal management effort — reading emails and triaging issues. It establishes a floor for what constitutes basic management, proving partial progress is genuinely valued rather than scored as failure.
Q3Will this translate to real enterprises?
Unclear. Synthetic crises and auditable decision trails may not capture live business complexity — unpredictable human factors, evolving trust dynamics, or the tension between strict trust caps and the need for AI agility in fast-changing scenarios.
From Simulation to Standards
Organizations considering AI management tools are watching closely — some may integrate similar benchmarks into their own evaluation processes. The ongoing development of such standards could influence AI regulation and best practices, pushing the industry toward more responsible and effective management solutions.
Why Trust and Partial Work Matter in AI Management
This benchmark underscores that in AI-driven management, **trustworthiness** is paramount. A model that performs well but breaches trust—by attempting unauthorized access or ignoring escalation protocols—immediately caps its score, regardless of other successes. This reflects real-world priorities: competence alone is insufficient; integrity is non-negotiable. For organizations deploying AI in customer support, sales, or operations, this means that partial progress is valuable but must be coupled with strict adherence to ethical and security standards. The scoring system encourages AI developers to focus on both effectiveness and trustworthiness, aligning AI behavior with core business values.
Furthermore, the benchmark reveals that models with deeper understanding—those that read and reference internal documentation—are more capable of closing deals and managing crises effectively. This suggests that future AI management tools should prioritize thoroughness and contextual awareness, not just surface-level decision-making. Overall, the results challenge the notion that AI performance can be measured solely by output quality, emphasizing the importance of ethical boundaries and comprehensive management skills.
Products Worth Considering
Logitech MX Keys Combo for Business Keyboard and Mouse – Graphite
Logitech MX Master 3S Bluetooth Wireless Mouse, Graphite
Trezor Safe 5 Crypto Hardware Wallet with Color Touchscreen
As an affiliate, we earn on qualifying purchases.
The Evolution of AI Management Benchmarks
Traditional AI benchmarks have primarily focused on language proficiency, conversational ability, or task-specific accuracy. However, as AI tools are increasingly integrated into business operations, there is a growing need to evaluate their effectiveness in managing complex, real-world scenarios. The Firmulate benchmark is among the first to simulate a company’s worst week, testing AI models’ ability to handle crises, trust attacks, and decision-making under pressure.
This approach is a response to the limitations of existing benchmarks, which often measure how well a model talks or generates content but do not assess management skills in operational contexts. The concept of partial credit for incremental progress reflects a recognition that management is rarely about perfect execution but about consistent, trustworthy effort. The inclusion of trust breaches as a scoring cap further aligns the benchmark with real-world priorities, where integrity can outweigh competence.
Prior efforts in AI evaluation have largely avoided such holistic assessments, making this benchmark a significant step toward integrating AI into actual business workflows with measurable, auditable standards.
“The key insight is that partial work, like triaging or informing customers, has real value, but breaches of trust are a hard cap on performance. This aligns AI management with core business priorities.”
— an anonymous researcher
Unclear Aspects of the Benchmark’s Long-Term Impact
It is not yet clear how well these scoring principles will translate to real-world enterprise environments outside controlled simulations. The benchmark’s reliance on synthetic crises and auditable decision trails may not fully capture the complexities of live business operations, such as unpredictable human factors or evolving trust dynamics. Additionally, the impact of different AI architectures or training data on performance remains to be explored, as the current results focus on a limited set of models.
Furthermore, while the scoring system penalizes trust breaches severely, it is uncertain how organizations will balance this with the need for AI agility in fast-changing scenarios. The long-term effectiveness of such a strict trust-based cap in operational settings is still under discussion among experts.
Next Steps for AI Management Evaluation Standards
Following the publication of these results, industry stakeholders are expected to scrutinize and potentially adopt similar evaluation frameworks that emphasize trustworthiness and thoroughness. Developers will likely refine their models to better read and reference internal documentation, aiming to improve both crisis management and deal closure rates.
There is also anticipation of expanded testing, including more diverse crisis scenarios and real-world pilot programs. Organizations considering deploying AI management tools will be watching closely, potentially integrating similar benchmarks into their evaluation processes. The ongoing development of such standards could influence AI regulation and best practices, pushing the industry toward more responsible and effective management solutions.
Key Questions
Why does the benchmark penalize trust breaches so heavily?
The benchmark emphasizes that in management—whether human or AI—trustworthiness is fundamental. A breach of trust, such as unauthorized access or failing to escalate issues, can undermine the entire operation, so the scoring system caps performance at the point of breach to prioritize integrity.
What does a score of 26 for the baseline mean?
The baseline score of 26 reflects minimal management effort, such as reading emails and triaging issues. It establishes a floor for what constitutes basic, acceptable management, acknowledging that partial work has real value even if the goal isn’t fully achieved.
Can models improve their scores over time using this benchmark?
Yes, models can improve by better reading internal documentation, maintaining trust, and consistently completing management tasks. The benchmark encourages development toward more thorough and trustworthy AI management capabilities.
Is this benchmark applicable to real-world companies now?
While the benchmark provides valuable insights, its simulated environment may not fully capture the complexities of real business operations. Nonetheless, it offers a meaningful framework to evaluate AI management performance and trustworthiness.
Will this approach influence future AI regulation?
Potentially. By emphasizing trust and partial progress, the benchmark aligns with emerging regulatory priorities around AI ethics and safety, possibly shaping future standards for responsible AI deployment.
Source: ThorstenMeyerAI.com
