Blog
Research

Confidential AI peer review before submission: what tools miss and wrongly flag

Choose an AI tool for confidential pre-submission peer review, then check its findings and look for omissions before sending your paper to a journal.

GZ

Guy Zana · Founder

November 13, 2026 · 5 min read

Share
Confidential AI peer review before submission: what tools miss and wrongly flag

In one test of AI reviews, every error that all tested systems missed was an omission. Before uploading your manuscript, check the tool's data terms. Then treat its report as a list of leads to investigate, including claims it may flag incorrectly.

Your decisionWhat to look forWhat to do next
Where to upload your draftTerms for your specific account and model, including training, retention, and third-party processingCheck your institution's rules and your coauthors' agreements before uploading
Which review to requestA report that identifies the manuscript pages behind each concernRead each cited passage and decide whether the concern applies
How to check claims about prior workAccess to the cited papers and passages, not just their titlesCompare each claim with its source and its stated limits
How to catch missing informationA checklist suited to your study design and journalCheck required disclosures and methods yourself

Check confidentiality before you upload

A promise not to train a model on your manuscript answers one question. It does not by itself tell you how long the provider retains the file, which other companies process it, or whether your institution permits the upload. Read the terms for the exact service and account you intend to use.

Check data handling before you choose a model. Ask whether you have permission to upload the manuscript and whether it contains participant data or material subject to a separate agreement. If the terms or your permissions are unclear, resolve that question before uploading.

Agent Bayes states in its terms that it does not use content or AI output to train its own general-purpose foundation models. The terms also describe the current training arrangements for OpenAI, Anthropic, and Google's paid Gemini API. Its privacy policy describes providers and processing. Read both for the full conditions.

AI reviews can miss omissions even when reports are pooled

In a 2026 preprint, Paul Litvak planted 100 errors across 10 open-access psychology papers and tested several models and two commercial review tools. The best single system found 71 of the 100 planted errors. Pooling all tested systems found 93.

The seven planted errors that no system found were omissions. One example was a removed disclosure about preregistration and whether all measures, manipulations, and exclusions were reported. A review cannot reassure you about a disclosure it never mentions.

Ask a separate question about what your manuscript leaves out. Use a reporting checklist appropriate to your study, and compare it against your methods, results, and disclosures. Ask a colleague to read those sections too.

The benchmark covers planted errors in psychology papers, not every kind of mistake in every field.

An AI judge matched review comments to planted errors, with a few spot checks. Litvak did not measure false positives or establish how often these error types occur in real papers. The scores are a reason to inspect the report, not an estimate of your paper's accuracy.

Different systems found different errors in that test. If your data permissions and budget allow another review, a different model may find something the first missed. More reports also mean more comments to check, and the seven omissions remained undetected even when the tested outputs were pooled.

A convincing flag can be wrong

One author reported on Reddit that Google's Deep Review identified real issues in a technical article but also flagged a Vision Transformer reference as likely hallucinated.

The author said the paper exists and the tool objected to an abbreviated author list. This is one user's account, not a measured error rate.

For each serious concern, locate the passage in your draft and check the evidence the report relies on. If the report disputes a reference, open the original publication and compare its title, authors, and claim. If the report cannot show why a point is wrong, investigate before changing your manuscript.

A review comment is a question to investigate, not a correction to accept automatically. Keep the comments that survive checking, document why you reject the others, and prioritize changes that affect interpretation before copyediting.

A real citation can still fail to support a claim

A manuscript can cite a real paper and still describe its findings too broadly. For example, a source may report an association in one population while a draft implies a causal effect in all populations. Open the source at the relevant result and compare the population, outcome, direction, and qualifications.

Agent Bayes can check mapped claims against papers in your library. Its support check is instructed to flag a missing qualifier and to say when a cited source may be missing from the library. A missing source in the library is a request to add or inspect that source, not proof that the citation is invalid. The claim-check walkthrough explains the process.

The check depends on the papers available to it and the passages it retrieves. It does not replace checking your data, statistical code, or the completeness of your methods. For a wider comparison of reviews, see AI vs human peer review.

Run a review of your own draft

Before submitting, add your manuscript to an Agent Bayes library and open it in the reader. In the Chat tab, select Peer review, add the journal's priorities, and send the prompt. Read the page citations behind each concern.

The walkthrough shows each step and reports a 1.1-credit example, your cost depends on the task.

For a separate check against prior work, add the cited papers to your library and run AI Peer Review Paper with Check support selected. This support check needs a paid plan.

The guide to reviewing a paper explains the referee report, and using AI to write a peer review covers the additional confidentiality question when the manuscript belongs to someone else.

New posts, straight to your inbox

No newsletter fluff, just an email when we publish something new.

Email me when a new post is published on the Agent Bayes blog. You can unsubscribe anytime. We'll first send a confirmation email, and we only use your details for this. See our Privacy Policy.

Enjoyed this? Share it.

GZ

Written by

Guy Zana · Founder

We are researchers and engineers building tools that help people reason over large bodies of literature without losing the thread back to the source.