AI peer review for your own paper: test every claim against your sources

A reviewer who knows your field will open a few of your references and check whether they say what you claim. Often, some of them do not.

In a study of highly cited biomedical papers, 39% of 3,063 citations had an accuracy problem, and 18% had a major one: the cited paper contradicted the claim, did not back it, or was not about it at all (Sarol et al., 2024).

So the most useful AI peer review of your own paper is narrow. Take every claim in the draft, find what your sources say about it, and flag the claims they do not back. This post runs that check on a real draft and shows what it caught.

Seven planted mistakes, seven catches

We wrote a two-page review called "Screen Limits and Adolescent Wellbeing". It cites eleven papers. Ten are open-access studies of social media and wellbeing, all in one Agent Bayes library. The eleventh is deliberately missing from the library. Then we planted seven mistakes of the kind reviewers find:

• an inflated number
• two reversed findings
• an association stated as a major risk
• a finding attributed to the wrong paper
• a citation to a paper that is not in the library
• a conclusion that goes further than the evidence

The check split the draft into 17 claims and tested each one against the library. Every claim got one of five verdicts and a score from 0 to 100, where higher means better backed:

• Supported, 80 or more: the passages back every part of the claim.
• Partly supported, 40 to 79: the core holds, but a number, population, or qualifier goes too far.
• Contested, 20 to 60: your sources disagree with each other.
• Contradicted, 25 or less: the passages, often the cited paper, say something incompatible.
• Not found in your library, 25 or less: nothing addresses the claim.

The check caught all seven planted mistakes.

The draft's closing recommendation, that schools should introduce daily screen limits, rests on the same conclusion and also came back Contested, at 48.

The abstract states the same overreaching conclusion more softly and came back Partly supported, at 66. None of the eight accurate claims came back Contradicted or Not found. The three plainest, such as the 4.5% figure itself, came back Supported at 95 to 100. The other five came back Partly supported at 60 to 76.

Accurate claims often come back Partly supported too, so read the issue text under each verdict. That is where the wrong-source flag appears.

Listing the claims cost 0.1 credits. Checking all 17 cost about 4 credits and took a few minutes.

Step 1: Turn the draft into a list of claims

A reviewer reads claim by claim, so the check starts with a list. In Agent Bayes, Map a paper's claims reads a paper from your library and creates a mindmap of its claims. A mindmap here is a tree of short statements: one branch per section of the paper, and one claim under it per sentence worth checking.

Each claim is rewritten to stand on its own, so "this effect" becomes the effect it refers to. The paper's own hedges stay, "may" remains "may". In-text references stay exactly as written, because they tell the next step which paper to look in.

The claims carry no citations, on purpose. A citation to the draft would make every claim look supported.

Step 2: Check each claim against the library

Select the claims and press F4 for AI Support Check. For each claim, the agent plans a few searches, one of them aimed at the paper the claim cites. It searches every library in the project, keeps the twelve most relevant passages, and judges them against the claim.

It judges only from those passages, not from what the model remembers. Then it writes the verdict and its score under the claim.

Each verdict is coloured from red to green by its score. Red and orange claims are the ones to fix first.

Step 3: Read the passage, then fix the sentence

Each verdict opens into the evidence. For the Collis and Eggers claim, the three contradicting passages all come from the cited trial itself, each with its page.

Under the evidence sit the issues, in plain words, and a suggested rewording that the passages do support. Here the rewording says the ten-minute limit had no significant effect on wellbeing or academic success, which is what the trial found.

Click a citation to open the page and read the sentence yourself before changing your manuscript. The verdict points you to the page, and you make the call.

The trap: your draft can vouch for itself

The obvious setup puts your draft in the same library as your references. The trouble is that the draft repeats every one of its own claims, word for word. A search for any claim finds the draft first. In a search tool without guards, that passage then counts as support.

A support check that can see your draft will find your draft, and a claim always agrees with itself. So the mindmap of claims remembers which paper it came from, and every check on it leaves that paper out of the search. The draft never takes up one of the twelve passage slots.

A second guard sits in the judge. A passage that only repeats a claim while citing someone else counts as second-hand, not as support. In a test with the draft left in the search, the Twenge claim still came back Not found in two runs, and the judge noted that the draft "repeats the claim but does not provide independent evidence for it."

If you build a similar check with your own tools, take the manuscript out of the search before anything else.

What the check cannot tell you

A support check tests your citations, not your science. It does not look at your statistics, your design, or whether your argument holds together.

It also only knows your library. "Not found in your library" means your sources cannot back the claim. Often the cited paper is simply missing, and the issue says so when it is. Add the paper and check again before you rewrite.

The judge is a language model reading passages, and it can misread one. Its suggested rewordings can also run long. Treat each verdict as a pointer to a page you then read, not as the last word.

Let Claude run the whole check

The same agents are available through the Agent Bayes MCP server. From Claude Code, a request like this lists the claims, runs the checks, and reports the weakest claims first:

Peer review my draft "Screen Limits and Adolescent Wellbeing". Map its claims, run a support check on every claim, and list the contradicted, contested, and not-found claims first, each with the page that settles it.

The verdicts land in the same mindmap, so the review stays where you can click through to the pages.

The step-by-step guide is in the docs: Mindmap and Review a Paper. Which of your citations would you least like a reviewer to open?