Which AI Model Setup Worked Best On My News Pipeline?

🔍 Read the full analysis: Which AI Model Setup Worked Best On My News Pipeline? on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Thorsten Meyer reports that GPT-6 Astra scored highest in his test of 71 AI model configurations for news writing, but he chose cheaper GPT-6 Luna for production. In results published October 13, Meyer said Astra scored 4.00 out of 5 in both English and German, while Luna cost about $0.0012 per article compared with Astra’s $0.12. The findings are from one publisher’s pipeline test, not an independent industry-wide evaluation.

Meyer tested configurations from September 8 to October 13, 2026, using the production prompt and real news sources. The test covered API, subscription, free-tier and locally run open-weight models. Each source was turned into an English and a German article, and two AI judges assessed source fidelity, fabrication, clarity about uncertainty, news value, structure and polish. German articles were also judged for language quality.

The initial test used two sources: a 6,340-character product filing and a 619-character political news item. Meyer says the shorter source better reflects much of his production work. He then ran the strongest configurations on 10 additional real sources, mostly short ones. A result counted only if the article produced valid output after the same repair step used in production.

In the follow-up run, Meyer reports that Astra averaged 3.93 across 12 articles, with a 95% bootstrap interval of 3.83 to 4.02. Luna averaged 3.67, with an interval of 3.59 to 3.76. Astra’s source-fidelity score in the standard test was 5.00; Luna’s was 4.71 in the robustness run, and all of Luna’s outputs were valid. Meyer estimates Astra took about 50 seconds per article, compared with Luna’s 17 seconds.

Quality Comes at a Steep Price

The results frame model choice as a quality, cost and speed trade-off, rather than a contest for the highest score alone. In Meyer’s test, Astra scored better, but cost roughly 100 times as much per article as Luna. For a high-volume publisher, that difference can shape whether a stronger model is practical for every story or reserved for selected work.

The test also highlights a risk that aggregate rankings can obscure: fallback models may generate unreliable copy when a primary model is unavailable. Meyer says one backup produced fabricated quotations and another did not produce valid articles in its four test attempts. In news publishing, invented quotes are a source-integrity problem, not merely a formatting defect.

These implications should be read within the test’s limits. The scores reflect Meyer’s prompts, sources, repair process and AI judges. They do not establish that Luna or Astra will perform the same way for other publishers, languages, article types or model versions.

How the Pipeline Was Tested

Meyer argues that general-purpose leaderboards do not directly answer which model should write articles for a particular publication. His test instead used the same prompt and workflow as his production pipeline, with source material provided to both AI judges. The judges were GPT-6.1 Sol, set to high reasoning, and Claude Opus 5.5; each scored the output from 1 to 5 across several editorial dimensions.

For the initial two-source test, Meyer set a rule that score gaps below about 0.3 points could be noise. He says the larger robustness run is a more dependable guide to ranking. He reports that Astra’s and Luna’s confidence intervals in that run did not overlap, while smaller differences among settings and models should not be overread.

One comparison challenged the idea that performance transfers between tasks. Meyer says Claude Opus 5.5, which he describes as a leading coding model, scored 3.46 in English and 3.39 in German as a news writer, with source fidelity of 3.75. Claude Sonnet 5.5 scored 3.67 and 3.61, but at an estimated $0.036 per article. These are Meyer’s measurements, not a general ranking of the models across all uses.

“A general intelligence score cannot answer the question I actually have to answer: which model should write the news articles my sites publish every day?”

— Thorsten Meyer, describing his test

Limits of One Publisher’s Test

The findings have not been presented here as an independent or peer-reviewed benchmark. The source material does not provide the full set of article outputs, judge-level scores, model prompts, or detailed results for all 71 configurations, so readers cannot independently reproduce every comparison from the reported figures alone.

The main test used only two source items before the follow-up run, and the robustness run covered 10 additional articles, mostly based on short sources. Meyer says confidence intervals support the Astra-Luna gap in that run, but performance on other beats, longer investigations, breaking news or different languages remains unclear. The assessment also depends on two AI judges, including Claude Opus 5.5, one of the models being evaluated.

The supplied account says Luna now runs first in the production route, but ends mid-sentence while describing what follows it. It does not establish whether a second fallback remains, what its identity is, or how the route behaves if Luna fails. Costs and latency are also reported for Meyer’s setup and may vary with usage and service pricing.

Production Use and Further Testing

Meyer says his pipeline now uses GPT-6 Luna at the default reasoning setting. He reports that Astra was ruled too expensive for routine news production on October 7, and that GPT-4.1 nano and locally run gpt-oss-120b were removed from the news route that day after poor test results. The account does not specify a future date for another benchmark or a planned model switch.

The practical next step for readers considering similar systems is to test models against their own source material and publishing requirements, including short, information-poor inputs. Meyer’s results suggest measuring factual fidelity and valid output alongside editorial scores, and checking failover models separately rather than assuming they match the main model’s quality. Whether Luna’s reported performance holds over a larger, longer production sample remains to be seen.

Key Questions

Which model scored highest in Meyer’s test?

GPT-6 Astra scored 4.00 out of 5 in both English and German in the standard test. Across the 12-article robustness run, Meyer reports an average of 3.93.

Why did Meyer choose GPT-6 Luna for production?

Meyer says Luna scored 3.67 in the robustness run and cost about $0.0012 per article, compared with about $0.12 for Astra. He selected Luna as the lower-cost option for routine production.

Did higher reasoning effort improve the articles?

Meyer found no reliable gain across Luna’s six tested reasoning settings. The maximum setting cost about 12 times more and took about 12 times longer than the default setting, without a score improvement beyond the test’s uncertainty.

What happened with the fallback models?

Meyer reports that GPT-4.1 nano scored poorly and generated fabricated quotes, while locally run gpt-oss-120b produced zero valid articles in four attempts. He says both were removed from the news route on October 7.

Can these results predict performance for other publishers?

Not on their own. The results come from one publisher’s prompt, sources, workflow and AI judges. Other newsrooms would need to test models on their own material and editorial standards.

Source: ThorstenMeyerAI.com