Blog
Research

AI vs human peer review: AI caught more planted errors, and still needs checking

AI models scored 53% to 96% at detecting planted major errors, against 27% for untrained human reviewers, and editors preferred AI reviews 26 of 30 times. What that means for checking your own paper.

GZ

Guy Zana · Founder

November 5, 2026 · 6 min read

Share
AI vs human peer review: AI caught more planted errors, and still needs checking

In a test with planted errors, AI models scored 53% to 96% at detecting the major errors in clinical trial reports. Untrained human reviewers scored 27%. In a second study, former journal editors preferred the AI review in 26 of 30 judgments. Both studies also show the limits of AI reviews, which suggests a practical way to use them: treat each concern as a question to check against the manuscript, not as a verdict.

The studies measure different things. One tests whether models catch errors deliberately inserted into clinical trial reports. The other compares reviews of manuscripts and checks whether statements in those reviews agree with the papers. Neither measures whether an AI review improves your own paper.

Researcher's questionWhat the studies showWhat to do
Can AI find mistakes before submission?Models scored 53% to 96% at detecting planted major errors. Untrained human reviewers in an earlier study scored 27%.Ask for specific concerns and check each against the relevant pages.
Can I trust every concern?Models also flagged error types in reports with no inserted errors. AI review statements were contradicted by the manuscript less often than human ones (5.5% vs 15.9%), but not never.Check the passage before changing a claim or analysis.
Does a polished report replace a specialist?Editors preferred the AI review 26 of 30 times, but they were not asked to read the manuscripts. Wording and factual accuracy are separate measurements.Ask a person who knows the field to assess context, significance, and missing work.

If a labmate is about to run their draft through an AI reviewer, send them this table first.

AI found planted errors, and also flagged problems in unmodified reports

In a medRxiv preprint, Brian and colleagues tested 15 language models on clinical trial reports with deliberately inserted major errors. On the three original reports, the models' major-error detection scores ranged from 53% to 96%, against 27% for untrained human reviewers in a 2004 study. The human reviews were older and scored differently, but on trials published in 2026, added to reduce the risk that models had memorized the reports, the models still scored 58% to 99%.

These are tests on clinical trial reports and planted errors, not a measurement of every problem in every research field.

Four of the five authors are affiliated with ResearchHub Foundation, whose platform rewards human peer review and is implementing AI-assisted review.

The authors also ran the models on control reports with no inserted errors. The models flagged 14% to 29% of the error types in those controls. The most common flag, in 576 of 704 control reviews, was that the conclusions were not supported by the results. A control without inserted errors can still contain other problems, so a flag is not automatically a proven mistake by the model.

For a researcher, the result changes how to read a review. Ask which sentence, table, or page prompted a concern. Then check whether the report describes the paper accurately and whether the proposed revision follows from that evidence.

Editors preferred the AI review 26 of 30 times, yet 37% of AI reviews had a flagged factual error

In a working paper, Gary King and Steven Worthington compared human reviews with AI reviews of 124 manuscripts. The human reviews came from a paid review service, which the authors run. The AI reviews used Claude Opus 4.8, with a web-enabled version used for the main quality comparison.

A separate AI model judged the reviews, and three former journal editors independently judged ten review pairs. Across 30 editor judgments, the editors preferred the AI review 26 times. The editors were not asked to read the associated manuscripts. Preference therefore says something about the reports they read, not by itself whether every point was correct.

The authors separately checked statements about the manuscripts, using AI reviews written without web access. An AI judge marked a statement incorrect only when the manuscript contradicted it. Statements that could not be verified remained in the denominator and were not counted as errors. The judge flagged 57 of 1,039 AI claim assessments, or 5.5%, and 152 of 956 human claim assessments, or 15.9%. At least one statement was flagged in 46 of 124 AI reviews (37.1%) and in 86 of 134 human reviews (64.2%).

A human check of a sample of 90 claims agreed with the AI audit on 87. Even a preferred review deserves a factual check. The study compares a particular set of manuscripts, reviewers, model, and judging procedure. Its percentages should not be treated as an error rate for every AI or human reviewer.

A specialist can notice what a report misses

An AI report can identify an apparent inconsistency in a table without knowing which overlooked work matters most to a field. A specialist can also disagree with another specialist. These differences matter because a useful review needs more than an inventory of possible errors. It needs judgment about which issues change the contribution and which revisions are feasible.

In one researcher's account of a rejected paper, the AI referee report was excellent, while the human referee's main catch was missing citations to the referee's own work. That is one anecdote, and it cannot establish how often either kind of reviewer finds missing references.

It does show why a reference request needs the same check as any other concern: does the requested work bear on your argument? Ask a knowledgeable reader what relevant literature or field-specific context is actually missing.

Check the concern against the page

A page reference makes a concern easier to investigate, but a page number alone does not prove the concern. Open the page, read enough context to understand the passage, and compare it with the review's description. If a review disputes a citation, open the cited source as well. A correct quotation can still be used to support an interpretation that goes too far.

For each concern, decide whether the paper contains the stated issue, whether the issue affects a conclusion, and what revision would address it. Keep the source passage with your decision. If the review gives no specific evidence, ask it to identify the relevant sentence or table before you act.

Use AI to generate questions, then use evidence and expertise to decide what to change. This also applies to a reviewer assessing someone else's manuscript. Check the journal's confidentiality and AI policies first. Some publishers prohibit uploading submitted manuscripts into AI tools. See our guide to reviewing a paper for an example and Elsevier's policy.

Try a page-cited review on a paper you can share

In Agent Bayes, add a paper you are allowed to analyze to a library, open it in the PDF reader, and select Peer review in the chat. The prompt asks for major concerns, with a page cited for each. Open the cited pages and decide which concerns warrant a revision. AI work uses credits, and the cost depends on the work performed.

For a separate check of whether your draft's claims match its cited papers, follow our claim-checking guide. In one small test described there, the check found all seven deliberately planted citation mistakes in a two-page draft.

New posts, straight to your inbox

No newsletter fluff, just an email when we publish something new.

Email me when a new post is published on the Agent Bayes blog. You can unsubscribe anytime. We'll first send a confirmation email, and we only use your details for this. See our Privacy Policy.

Enjoyed this? Share it.

GZ

Written by

Guy Zana · Founder

We are researchers and engineers building tools that help people reason over large bodies of literature without losing the thread back to the source.