How accurate are AI research assistants? Agent Bayes on three public benchmarks

Agent Bayes answered scholarly questions more accurately than every published result on IRPAPERS, gave zero wrong answers on LitQA2, and scored 5 of 5 for factual correctness on ResearchQA. All three benchmarks hand an assistant a set of papers and ask questions about them, which is how you use Agent Bayes with your own library.

Why accuracy comes first

An AI answer is only worth reading if its sources hold up. Artificial Science's review of research assistants describes tools with problems in their references and gaps in their searches.

This month a Springer Nature journal retracted an article over references that were real but did not support the sentences citing them, a case Papex walks through in its citation-checking guide.

So we measured two things: whether the answers are right, and whether the cited passages contain the evidence.

The scorecard

We indexed the papers each benchmark supplies and asked every question once through library chat, on gpt-6.1-sol at effort Medium. Answers were scored with each benchmark's official scoring method.

IRPAPERS: ahead of a system handed the right page

IRPAPERS asks 180 questions over 166 information-retrieval papers. The papers were chosen so that many pages share vocabulary and methods, which makes the one page holding the answer hard to find.

The benchmark's gpt-4.1 judge accepted 97.2% of Agent Bayes answers, with a 95% confidence interval from 93.7% to 98.8%. The best published configuration scored 82%. A published configuration that was handed the correct page in advance scored 74%.

Library chat does not stop at one search. It searches the library, reads the pages it finds, and searches again when the evidence is thin, so the answer does not depend on the first search result being the right page.

LitQA2: zero wrong answers

LitQA2, introduced with PaperQA2, asks multiple-choice questions whose answers sit deep in the full text of biology papers. We put the 69 open-access papers that answer its questions into one library.

Agent Bayes answered 61 of 67 questions correctly, and not one answer was wrong. On five of the remaining six, it said the papers did not settle the question.

That zero matters more than the 91%. A wrong answer reads exactly as confidently as a right one, so an assistant that says "not enough evidence" keeps you from building on a guess. The 91% accuracy is level with VecTree-RAG, the best published system that also answered from a library of open-access papers.

ResearchQA: perfect factual scores on single-paper questions

ResearchQA 2026 asks questions about one paper at a time. On a 20-question sample, the benchmark's Gemini judge gave every Agent Bayes answer 5 of 5 for factual correctness, and 4.95 of 5 on average for completeness. The best published scores are 5.0 and 4.98.

The citations held up as well. Agent Bayes cites whole passages, so we checked whether each cited passage contains the evidence quote the benchmark expects, in full or its opening portion. On average, 85% of an answer's cited passages contained the expected evidence, and 97% of the evidence sections an answer needed had a matching citation.

Under 3 credits and 14 seconds per answer

Each answer cost under 3 credits and arrived in about 14 seconds at the median. Effort Medium was enough: raising the effort level cost 1.7 times as much and scored no higher.

If you are choosing a model for this work, our Sol, Astra, and Opus comparison and Argon and Opus comparison test the models side by side on the same research questions.

Try it on your own papers

Benchmarks use someone else's papers. The test that counts is yours. Add your PDFs to a library, or sync them from Zotero, then open your project's Chat page and ask the question you are stuck on. Every claim in the answer cites a passage from your papers that you can open and check.

Already have a draft? AI peer review for your own paper checks each of its claims against your library. To set up your first library, follow the quickstart.