🔍 Read the full analysis: FrontierCode: A Maintainer’s Perspective On AI-Generated Pull Requests on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Artificial Analysis evaluated 23 models on FrontierCode, a benchmark that scores whether maintainers would accept AI-generated code changes, not just whether tests pass. Claude Opus 5.5 ranked first, but passed every blocking check in 57.7% of attempts; results also show large differences in cost and performance.
Artificial Analysis published results on October 12 for 23 models tested on FrontierCode, a coding benchmark that asks whether an AI-generated pull request meets maintainer standards. Claude Opus 5.5 ranked first with a 52.2% score, but cleared every blocking check in 57.7% of attempts, indicating that even the top result does not establish reliable, unattended merging.
FrontierCode’s results distinguish between passing tests and producing a change that reviewers could accept. Its score combines checks for correctness, regressions, test quality, scope and code review. Failing any blocking check gives an attempt a score of zero; passing all blocking checks yields a score from 70% to 100%, depending on additional quality checks. The benchmark also penalizes oversized changes and edits outside the relevant files.
Artificial Analysis ran each model using the same open-source agent harness, Stirrup, with a limit of 250 turns per attempt and three attempts per task. The review stage used GPT-6 Luna (medium) as the judge. On the published Main set, Claude Opus 5.5 (max with fallback) scored 52.2%, followed by GPT-6.1 Sol (max) at 49.1% and Claude Fable 5.1 (max with fallback) at 49.0%. The leader’s 57.7% pass rate refers to attempts clearing all blocking checks, not to a guarantee that a patch would be accepted in every real-world repository.
Performance and price varied substantially. Artificial Analysis reported a cost of $10.15 per task for Opus and $0.81 for GPT-6.1 Sol. Xiaomi’s MiMo-V2.6-Pro scored 45.4% at $0.18 per task, while GPT-6 Luna (max) scored 41.9% at $0.09. The source says nine of the 100 tasks in the Main set were not solved by any model on any attempt.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Maintainer Checks Change the Ranking
The benchmark addresses a gap in coding evaluations: a patch can pass tests and still be difficult or risky to review. FrontierCode applies checks for whether tests meaningfully detect errors, whether a change stays within the requested scope and whether it meets maintainers’ review criteria. That makes its results relevant to teams considering AI agents for repository work, where review effort and unintended changes matter alongside functional correctness.
The scores also suggest that a model’s general capability ranking may not predict its performance on review-oriented coding tasks. In the comparisons supplied by Artificial Analysis, Claude Sonnet 5.5 placed second among the listed models on the Intelligence Index but fifth on FrontierCode. Grok 4.7 ranked 18th on FrontierCode, while DeepSeek V4.1 Flash and GPT-6 Luna performed better there than their listed general-intelligence positions might suggest. These are benchmark comparisons, not evidence that a model will perform the same way across every codebase or workflow.
Cost is another practical distinction. The reported per-task figures range from $0.09 to $12, depending on the model and configuration. A higher score can come at a much higher price: Artificial Analysis puts GPT-6.1 Sol at 49.1% for $0.81 per task and Opus at 52.2% for $10.15. Those figures describe the benchmark setup; they do not include every possible cost of deploying an agent or having people review its work.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Tests Pull Requests
FrontierCode was developed by Cognition with open-source maintainers and announced in June 2026, according to the source material. Version 1.1 followed in July with rules intended to prevent agents from looking up task solutions online. More than 20 maintainers created tasks based on 36 repositories, including Celery, Budibase, uppy and Mattermost. The source says maintainers spent more than 40 hours on each task.
The full benchmark contains 150 tasks, while Artificial Analysis evaluated models on a 100-task Main set. The tasks are private, so models could not have encountered their solutions during training, according to the benchmark description. Checks include hidden tests, command checks, reverse tests that assess whether an agent’s tests fail on the original code, adaptive tests for complex behavior, limits on changed files and diff size, and an LLM review against maintainer criteria.
In Cognition’s announcement, Celery maintainer Tomer Nosrati described the approach this way: “Where others grade like a CI, FrontierCode grades like a tech lead.” The remark captures the benchmark’s focus on review judgment, but the published scores remain evaluations under a defined harness and task set rather than direct measurements of every team’s production workflow.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, Celery maintainer, in Cognition’s announcement
automated pull request reviewer software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Limits of the Published Results
The results do not show how each model would perform across all repositories, task types or review teams. The source describes a 100-task evaluation set and a particular agent harness, turn limit and review judge; changes to those conditions could affect outcomes. The benchmark’s scores also do not establish that a patch passing its checks is safe to merge without human review.
Several models had no result as of October 13, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their absence means the published table is not a complete comparison of every model mentioned in the source. It is also unclear from the supplied material whether or when those evaluations will be added. The figures are reported by Artificial Analysis and the benchmark developers; independent replication details were not provided in the source.
AI-powered code quality analysis tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Awaiting More Model Results
The next point to watch is whether Artificial Analysis adds results for the models not listed and whether future benchmark runs change the rankings. Any new results would need to be compared using the same task set and evaluation conditions to support a direct comparison. For developers and maintainers, the current scores offer a basis for testing candidate models in their own repositories, but the benchmark does not settle whether a particular system can safely handle pull requests without human review.
software development automation tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What does FrontierCode measure?
FrontierCode evaluates AI-generated code changes against checks for correctness, regressions, test quality, scope and maintainer-style review criteria. It is designed to assess more than whether a patch passes ordinary tests.
Which model ranked first?
Claude Opus 5.5 (max with fallback) ranked first in Artificial Analysis’s results with a 52.2% FrontierCode score. It cleared every blocking check in 57.7% of attempts.
Does the benchmark show AI pull requests are ready to merge automatically?
No. The results show that even the leading model failed to clear all blocking checks in a substantial share of attempts. The benchmark does not establish that automated changes can be merged safely without human review in real repositories.
How much did the tested models cost per task?
Reported costs ranged from $0.09 per task for GPT-6 Luna (max) to $12 for Claude Fable 5.1. The source lists Claude Opus 5.5 at $10.15 and GPT-6.1 Sol at $0.81 per task under the evaluation setup.
Were all major models included?
No. As of October 13, the source listed no FrontierCode result for Gemini 4 Argon, Muse Wonder, Muse Wonder Full or Mistral Large 4 Preview. The timing and availability of future results were not specified.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
