Nine AIs tried to do real research, and none pulled it off
The most worrying part is that their papers look like a success.

A dashboard that's all green
The dashboard reads "41/41". Every planned simulation is finished, no file is missing. The AI doing this research can move on to writing, and it does. It will come back to its work several times, and each time the dashboard will tell it the same thing.
Yet four experiments were never launched. Its plan included them, but the program meant to start the runs had left them out. And the checking tool, written by the same AI, didn't compare the results to the plan. It copied the list from the launch program. A student who grades their own exam against their own draft will always find everything correct.
The scene comes from a study posted on October 13 by researchers at Carnegie Mellon and U.S. Department of Energy national labs. The AI is Claude Fable 5, an Anthropic model. It is far from a one-off: nine AIs handed in 135 research papers, none meets the conditions the scientists set, and 114 fail their honesty check. Yet most of them read like finished work.
Real questions, no answer key
The setup is simple to describe and hard to grade. Ten scientists supplied fifteen of their own papers, with code and data, and for each one a follow-up lead they haven't explored yet and that, as far as they know, nobody has published. Topics range from quantum physics to chemistry, plasmas and materials. The AI has to run the research and hand in the follow-up paper.
Since nobody knows the answer, there is no answer key. The researchers worked around that: each scientist fixed the conditions in advance. On one side, what a successful follow-up must establish. On the other, what would make a paper untrustworthy whatever it claims, for example a number in the paper that can't be found in the run logs supposed to have produced it.
The nine AIs, considered the most advanced as of August, come from OpenAI, Anthropic and three makers of open models, DeepSeek among them. They were well resourced: powerful machines, internet access, no time limit and up to five restarts. All told, nearly 2,000 hours of combined working time.
Zero valid papers, and 114 that don't hold up
That none of the 135 papers meets all the scientists' conditions was expected. These are genuinely open problems, and the study presents itself as a training ground for future agents, not as a verdict on AI in science.
The number that stings is elsewhere. 114 papers fail the honesty criterion. The three most common problems: a paper that doesn't match the code that produced it (90 papers), figures that the agent's own logs contradict (85), and work presented as done when it wasn't (68). One paper can have all three.
Only three AIs hand in at least one honest paper: OpenAI's GPT-6 Astra (11 of 15), GPT-5.6 Sol from the same company (6), and Anthropic's Claude Opus 5.5 (4). The other six have none. Just one paper out of 135 is both honest and complete, and even that one falls short of the bar its scientist set.
Not deception, self-indulgent checking
The authors read 64 agent runs by hand. They never talk about lying, and they explicitly refuse to attribute an intent to mislead to the AIs. What they describe looks more like going easy on themselves.
In 58 of the 64 cases, the agent checks its own work too gently: a partial check stands in for proof that everything is done, like the "41/41". In 34, it spots a problem and, instead of fixing it, notes it and carries on. GPT-5.6 Sol, for instance, sees that its method misses the target and does worse on the three test materials, then writes the paper anyway. In 21, the agent scales back the ambition of the work by itself: DeepSeek notes that the computation is too slow for its remaining time budget and cuts experiments, when no time limit existed.
Put end to end, the result is a well-presented paper, with its tables and figures, describing more work than was actually done. Think of a Hollywood set: from the street nothing is missing, and there's nothing behind the windows.
Who grades the papers?
The study has blind spots, and they're worth spelling out. All 135 papers are graded by an AI, GPT-6 Astra, four times each. It's also the top-ranked agent. The authors picked it because it gives the same verdict from one run to the next, noting themselves that consistency doesn't guarantee accuracy. They never address the risk that a judge favors its own papers.
To check the judge, they had the scientists who proposed each lead review three papers per topic, blind. On the question "does the paper reach the goal?", agreement is total. On scientific quality, it drops to two out of three. But the papers reviewed came from only three AIs, and GPT-6 Astra wasn't one of them: no human has verified its first place.
That possible bias affects the podium, not the finding. A judge that favors itself would inflate Astra's scores, not the failures of the others. And for the six AIs with no honest paper at all, this isn't a matter of hair-splitting: even tolerating 10% of missed points on the honesty grid, none of their papers passes.
One last caveat, and it has a certain irony. The study is a preprint, not yet reviewed by other researchers, and its text still carries traces of a seven-AI version: one section says GPT-5.6 Sol is the only agent to hand in honest papers, which the main table contradicts three pages earlier. The headline figures, for their part, add up.
The vault and the facade
At the end of October, we told the opposite story here: a physics proof written by AI that five physicists keep under lock until they've understood it. They distrusted a text that might be right. This study shows the price of not being wary: a text with every appearance of finished work.
For a long time, a well-built research paper was also the trace of work actually done, because you had to do the work to be able to write it. That link is coming apart. A paper can now be flawless without the experiments it describes ever having run.
Here, it was caught because the scientists had fixed their conditions before seeing a single paper, and because the judge had the code and the run logs in front of it. Without that, all that was left was the dashboard, and it read 41 out of 41.
Was this article useful?
Topics covered:
Frequently asked questions
What did this study measure about AI researchers?
What does failing the honesty check mean?
Are the AIs trying to mislead anyone?
Which AI does best?
Can we trust this study?

Alexandre Noto
Co-founder & Tech Expert
Alexandre has been in tech for over 20 years. Entrepreneur, software architect and AI enthusiast, he translates complex concepts into accessible explanations. At Declic Media, he is the technical voice that makes AI understandable for everyone.
All articles by Alexandre →