Can I use AI for screening in a systematic review?

Screening is the part of a systematic review most obviously suited to automation and the part where an error is hardest to detect. A false include costs you fifteen minutes at full-text stage. A false exclude removes a study from the review permanently and leaves no trace, because nobody audits the records that were dropped.

That asymmetry should drive the whole design. Use the tool in the direction where errors are recoverable: as a second screener alongside a human, as a prioritizer that orders records so the relevant ones surface first, or as a flag for records a human should look at twice. Do not use it as the only thing standing between a record and exclusion.

The reproducibility requirement is the other constraint. A systematic review is defined by the fact that another team could repeat it. A screening step you cannot describe precisely enough to be repeated undermines the design, whatever its accuracy, which is why the reporting matters as much as the validation.

None of this is an argument against automation. Screening thousands of abstracts by hand introduces its own errors through fatigue, and a well-validated automated step used as a second screener can be more consistent than a tired human. The requirement is evidence, not abstention.

The human baseline is worse than most students assume. Across 329,332 abstract screening decisions from 25 systematic reviews, nine clinical areas, and 85 reviewers, the total human error rate was 10.76 percent, ranging from 5.76 percent to 21.11 percent between reviewers (Wang et al., 2020, p. 2). The same analysis puts abstract screening at about 0.9 minutes per reference and full-text screening at 7 minutes, so a search returning 1,000 studies costs roughly 952 hours of work (Wang et al., 2020, pp. 4, 8). The authors argue that an algorithm with error rates similar to human reviewers may be adequate precisely because the human gold standard is itself imperfect (Wang et al., 2020, p. 6).

The workload savings are real but domain-specific. In a review of HIV impacts on female sex workers, a random forest classifier trained by distant supervision cut manual reading by 80 percent, and a sensitivity analysis showed that training on only 10 percent of the data still let the classifier skip 75 percent of future documents while retaining 80 percent of the relevant ones (Li et al., 2022, p. 1). Treat that as one case study, not a guarantee for your review.

What do reporting guidelines require me to declare?

Which stage the tool was used at, its name and version, the criteria it applied, whether a human checked every decision or a sample, and the agreement between the tool and the human screeners. Anything a reader would need to judge whether your included set is trustworthy.

PRISMA asks for enough methodological detail that the review could be reproduced, and an automation step is part of the method. Where your target journal has its own AI disclosure requirement, that applies as well, and the two statements should not contradict each other.

Report the numbers rather than a reassurance. "Agreement was 94 percent on a sample of 500 records, with three relevant records excluded by the tool and recovered by the human screener" is a sentence a reader can evaluate.

How do I validate an automated screening step?

Screen a random sample of several hundred records both ways and report the agreement, including how many relevant records the tool would have excluded. Recall matters far more than precision here, because a missed study is invisible in the final review and a false include is caught at full text.

Validate on your own records rather than copying a reported accuracy figure, because performance moves with the corpus and with what you feed the model. On an imbalanced dataset, a baseline classifier reached precision 0.268 and recall 0.645, while a random forest over refined clusters raised recall to 0.731 at the same precision (Li et al., 2022, pp. 7-8). Adding metadata and citation features beyond the title and abstract improved XGBoost accuracy by about 3 percent on a 1,697-citation HPV corpus and about 2 percent on a 2,865-citation pediatric pneumococcal corpus (Du et al., 2024, pp. 1, 5). Transformer models beat conventional models when only the title and abstract were available, but the conventional models became comparable once citation features were supplied, and at lower computational cost (Du et al., 2024, p. 6). Newer and larger is not automatically better.

Two qualifications sit behind those numbers. The models in Du et al. received limited task-specific customization and were evaluated mainly on health economics and outcomes corpora, so the authors state that generalizability to another field needs formal examination, and they list explainability, bias, and hallucination as unresolved concerns for large language models like ChatGPT and Gemini (Du et al., 2024, p. 6). Wang et al. are more permissive about accepting algorithmic error, so the two papers set different bars. Your protocol has to pick one and say which.

Do the validation before you rely on the tool, not afterwards. Validating at the end tells you whether the review was sound and gives you no opportunity to fix it if the answer is no.

If recall is not high enough for your comfort, the tool can still earn its place as a prioritizer. Ordering the records so the likely includes come first makes human screening faster without removing the human decision, and that use needs much weaker guarantees.