Using AI to peer review a paper: a page-cited referee report in three minutes

An AI referee report is worth reading only when every concern points to a page you can open. Without the page, a concern is a guess you cannot check, and a fluent guess can be worse than none because it sounds like expertise.

We asked Agent Bayes for a referee report on a 25-page open-access experiment and checked every point against the paper. The report took under three minutes. Its three major concerns were, in our reading, the ones a careful methods reviewer would raise, and each one cited the pages it rested on.

First, check whether you are allowed to

If someone sent you the manuscript to review, it is confidential.

Elsevier's policy says reviewers "should not upload a submitted manuscript or any part of it into an AI tool", because it may violate the authors' confidentiality and proprietary rights (Elsevier). Many journals and funders have similar rules.

For a manuscript under review, follow the journal's policy, and ask the editor when it is unclear. Everything below is about papers you are free to analyze: published articles and preprints, for a journal club, a literature review, or practice before you review for real.

The paper and the prompt

The paper is Walsh et al. (2024), "Does putting down your smartphone make you happier?", published open access in PLOS ONE. It reports a preregistered experiment in which 338 Gen Z university students restricted their smartphone use, their social media use, their water use as an active control, or nothing, for about a week.

The setup takes three steps. Add the PDF to a library, open it in the Agent Bayes PDF reader, which has a chat panel beside the page, and pick the ready-made chat prompt Peer review. The prompt asks for the report a journal referee would write, in five parts:

1. a one-paragraph summary of the paper and its contribution
2. major concerns that affect the validity of the conclusions, with what the authors should do
3. minor concerns about clarity, presentation, and missing detail
4. requested revisions
5. a recommendation, from accept to reject, with a justification

It ends with one instruction that matters more than the rest: cite the page for each concern.

What the report found

Before writing, the agent read eight of the 25 pages, from the abstract through the discussion, and searched the rest. It recommended a major revision, and its major concerns were these.

Exclusions after randomization. The analysis drops 33 participants in the restriction groups because their reported use went up during the week. The report pointed out that excluding people based on what happened after assignment weakens the point of randomizing.

It asked for an intention-to-treat analysis that keeps everyone, and cited page 7, where the exclusions are described.

Many outcomes, few corrections. The paper tests many outcomes across several group comparisons, and some effects sit near the significance threshold. The report asked which outcomes were primary and how false positives were controlled. It cited pages 12 and 14.

A title broader than the results. The digital media group was not significantly happier than both control groups, and the discussion says neither restriction improved emotional wellbeing. The clearer gains were in life satisfaction, mindful attention, and self-esteem. The report asked the authors to say so in the abstract and in how the title frames the result. It cited pages 14 and 16.

The minor concerns were smaller and also checkable: screen time was measured only on iPhones, the intervention lasted seven to thirteen days rather than a fixed week, and the sample came from one university participant pool and was mostly women.

How to check an AI referee report

Treat the report as a list of leads. For each concern:

1. Click the citation and read the passage. A concern you cannot find on the page does not go into your report.
2. Ask whether the concern matters for this paper. AI reviewers often flag multiple comparisons in studies with many outcomes. Whether it changes the conclusion is your judgement.
3. Look for what is missing. The report reads the paper on its own terms. It cannot tell you that a better study on the same question exists, or that the field has moved on.

We opened the pages behind each concern in this report, and each one matched. The paper describes the 33 exclusions on page 7, and nowhere mentions a correction for multiple comparisons. That is the bar to hold any AI review to.

Where an AI referee report falls short

It knows only the paper in front of it. It cannot judge novelty against the field, and it will not notice a missing citation to the study everyone in your area knows.

It does not rerun anything. It reads the reported analysis, not the data or the code, even when, as here, the data are public. This report did not touch the open data at all.

It separates major from minor concerns, but within each list every point gets the same confident tone. Deciding which concern actually changes the verdict is your job.

Going further: check the paper's claims against your library

A referee report tests the paper on its own terms. To test what it says about other studies, Agent Bayes can list the paper's claims and check each one against the papers in your library, with a verdict under each claim. That check is described in AI peer review for your own paper, and it works just as well on someone else's paper.

The steps for the referee report are in the docs: Peer Review a Paper. Which concern would you have added that the report missed?