🔍 Read the full analysis: FrontierCode: Would Your AI’s Pull Request Pass A Maintainer’s Review? on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Artificial Analysis reported results for 23 models on FrontierCode, a benchmark that scores coding agents on whether maintainers might accept their patches, not just whether tests pass. Claude Opus 5.5 led with a 52.2% benchmark score and cleared every blocking check in 57.7% of attempts; the results indicate that human review remains important, while model costs vary sharply.
Artificial Analysis published results for 23 AI models on FrontierCode, a coding benchmark designed to judge whether a patch would meet maintainers’ review standards rather than merely pass tests. Claude Opus 5.5 ranked first with a 52.2% score, but cleared all blocking checks in 57.7% of attempts, suggesting that even the top result does not make human review unnecessary.
In the evaluation, GPT-6.1 Sol placed second with a 49.1% score and a 54.7% blocking-check pass rate. Claude Fable 5.1 followed at 49.0%. Artificial Analysis ran its 100-task Main set using the same open-source agent harness, Stirrup, for each model, allowing up to 250 turns per attempt and three attempts per task. The review stage used GPT-6 Luna (medium) as the judge.
FrontierCode assigns scores based on six kinds of checks: hidden tests, command checks for regressions and cleanliness, tests that must fail against the original code, adaptive tests, limits on files and diff size, and an LLM review against maintainers’ criteria. A failed blocking check gives an attempt a score of zero. Passing all blocking checks can yield a score from 70% to 100%, depending on other quality checks. Attempts flagged by Cognition’s trajectory scanner for looking up solutions also receive zero.
The benchmark’s score and pass rate describe different things: the 52.2% score reflects the overall scoring system, while the 57.7% pass rate is the share of Opus attempts that cleared every blocking check. Artificial Analysis reported nine of the 100 tasks were not solved by any model on any attempt. Results are therefore about this benchmark setup, not a guarantee of how a model will perform in every repository.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Review-Ready Patches Remain Difficult
The results put a number on the gap between producing code that runs and producing a patch that a maintainer might accept. More than four in ten Opus attempts failed at least one blocking check in this evaluation. That does not establish that all AI-generated changes need the same level of scrutiny, but it does argue against treating test success alone as proof that a pull request is ready to merge.
The benchmark also shows why model selection can depend on the task and budget. Artificial Analysis listed MiMo-V2.6-Pro at 45.4% for $0.18 per task and GPT-6.1 Sol at 49.1% for $0.81, compared with Opus at 52.2% for $10.15. Those figures are the source’s reported costs per task; they do not by themselves establish total deployment costs or value across different workloads. The relatively close scores among several leading models make cost and review requirements relevant alongside rank.
FrontierCode’s maintainer-focused checks may also reveal differences that broad capability rankings miss. In the supplied comparison with Artificial Analysis’s Intelligence Index, Claude Sonnet 5.5 ranked second among the listed models on that index but fifth on FrontierCode; Grok 4.7 ranked 18th on FrontierCode. General capability scores do not directly predict patch acceptability under this benchmark’s criteria.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Grades Code Changes
FrontierCode was developed by Cognition with open-source maintainers and announced in June 2026; version 1.1 followed in July with rules against finding solutions online. More than 20 maintainers created tasks drawn from 36 repositories, including Celery, Budibase, uppy and Mattermost. According to the source material, maintainers spent more than 40 hours on each task. The complete benchmark has 150 tasks, while Artificial Analysis evaluated the 100-task Main set.
The tasks are private, a design intended to reduce the chance that models have encountered their exact solutions during training. Its tests also target review concerns: for example, reverse tests check that an agent’s new tests fail on the original code, while scope checks can penalize changes to unrelated files or excessively large diffs. The benchmark’s approach was summed up by Tomer Nosrati, a Celery maintainer, in Cognition’s announcement: “Where others grade like a CI, FrontierCode grades like a tech lead.”
The leaderboard’s leading group was close: nine models from six labs scored between 43% and 52%. The source identifies MiMo-V2.6-Pro as the highest-ranked open-weights model at 45.4%, followed by DeepSeek V4.1 Flash at 44.0% and Kimi K3 at 43.6%. These are benchmark results under the stated harness and settings, not universal measures of coding ability.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, in Cognition’s announcement
software development code review software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Limits of the Published Results
The reported scores cover one benchmark, one 100-task evaluation set and specified model configurations. The source does not establish how closely these tasks represent every project’s review process, nor whether the scores will predict outcomes across different languages, repositories or agent setups. A score is not a direct estimate of the share of real-world pull requests that would be merged.
Several models had no result as of October 13, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview, according to the source. Their absence means the published ranking is not a comparison of every prominent model. The source also does not provide enough detail here to independently evaluate the review judge’s consistency or how results might change with different harness settings.
As an affiliate, we earn on qualifying purchases.
More Models May Join the Ranking
The next useful development will be results for models that were missing from the October 13 leaderboard, along with future benchmark runs that make comparisons across model versions possible. The source specifically points to Muse Wonder Full as a model to watch, but supplies no FrontierCode result or publication date for it. No release schedule for additional evaluations is given.
For now, readers can use the published table as evidence about performance under FrontierCode’s task and scoring rules, while treating human review as part of the process. Further runs could clarify whether the close scores near the top persist and how rankings shift as models and evaluation methods change.
machine learning code analysis software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is FrontierCode?
FrontierCode is a coding benchmark developed by Cognition with open-source maintainers. It evaluates code changes using tests, scope and cleanliness checks, and an LLM review against maintainers’ criteria.
Which model ranked first?
Claude Opus 5.5 (max with fallback) led the Artificial Analysis results with a 52.2% FrontierCode score. It cleared every blocking check in 57.7% of attempts.
Does a 52.2% score mean Opus passed 52.2% of attempts?
No. The 52.2% figure is its overall benchmark score. The separately reported blocking-check pass rate was 57.7%.
Did any model solve every task?
No model solved nine of the 100 tasks on any attempt, according to the source material. That finding applies to the evaluated models and the benchmark’s reported runs.
Does this benchmark show that AI pull requests can be merged without review?
No. The results do not prove how every real-world pull request will fare, but even the leading model failed at least one blocking check in many attempts. The benchmark does not support treating test success alone as a reason to skip review.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
