Blog
Research

ChatGPT deep research vs dedicated academic research agents: what AstaBench measured

ChatGPT deep research vs dedicated academic research agents: a 2025 benchmark compared their cited answers to computer science questions. Learn what the scores mean for checking sources.

GZ

Guy Zana · Founder

November 7, 2026 · 5 min read

Share
ChatGPT deep research vs dedicated academic research agents: what AstaBench measured

In a 2025 test of research reports, Asta Scholar QA, Elicit, and SciSpace scored higher than OpenAI Deep Research on the same 100 computer science questions. The AstaBench study averaged four measures of answer relevance, answer coverage, and citation support into a score out of 100.

Its results help explain what to look for in a research tool, but they do not establish which tool is best today.

Your taskWhat the evidence suggestsA practical choice
Get an overview of an unfamiliar questionThe tested academic tools scored 84.6 to 88.7 out of 100, compared with 79.4 for OpenAI Deep Research.Use a report to find directions and papers to investigate. Treat the 2025 ranking as historical.
Check a claim you might citeA score averages performance across reports and citation measures. A particular sentence needs its own source check.Open the cited source and compare the claim with the passage in context.
Work with papers you already selectedThe benchmark compared systems with different search sources and interfaces.Search your own library when the question depends on a specific collection of papers.

What AstaBench asked the tools to do

The Allen Institute for AI researchers built ScholarQA-CS2 to test long answers to scientific questions. Each system received a question and produced a report with citations. The test set contained 100 computer science questions, including real questions from users. One example asks how researchers evaluate diversity in recommendation systems.

The researchers used the same Gemini 2.5 Flash judge to assess four aspects: answer coverage, paragraph relevance, how often claims had supporting citations, and how often citations supported their claims. The published score is the average of those four measures, expressed out of 100. It is a composite report score, not the percentage of sentences proven correct.

The authors compared those automated assessments with expert judgments as part of their validation. An automated assessment remains an estimate, and the questions cover one subject area. The score is useful for comparing these tested reports, not for predicting accuracy in every discipline.

The 2025 scores favored three academic tools

The following results come from Table 13 of the original AstaBench paper. The intervals are the authors' 95% confidence intervals around each mean score.

System testedScore out of 100
Asta Scholar QA with tables, using o388.7 ± 1.2
Elicit85.5 ± 1.6
SciSpace Deep Review84.6 ± 1.3
OpenAI Deep Research79.4 ± 1.4
Perplexity Sonar Deep Research67.3 ± 1.2

The systems did not all search the same collection. The paper describes Elicit searching Semantic Scholar and SciSpace searching scholarly indexes, while the tested OpenAI system used web search. AstaBench attributes the three leading academic tools' advantage to their citation subscores. The comparison does not isolate search source as the cause.

The result describes specific 2025 systems, not a current product ranking. Elicit submitted its answers on 3 April 2025, and SciSpace submitted its answers on 13 June 2025. The OpenAI reports were generated through the API using o3-deep-research, with the paper's figure identifying the 26 June 2025 model version. That API test is not a direct test of today's ChatGPT interface.

The Asta team built both the benchmark and Asta Scholar QA. The authors also worked with commercial providers where possible. The paper supplies the method and results, so readers can inspect the comparison and its limits.

A report score and a source check answer different questions

A report can cover a question well while one sentence overstates a paper. For example, a paper may report an association in one population, while a summary describes a causal effect across all populations. The paper exists and the citation may lead to it, but the sentence asks more of the evidence than the authors found.

AstaBench includes citation measures in its score, so the benchmark does assess attribution. A combined score still cannot tell a reader whether the particular sentence they want to use is supported. For that decision, open the source and read the passage around the result, including its population, measurement, and qualifications.

Ask three questions of a citation before using a claim:

  1. Does the cited source actually contain the result, or does it report another paper's result?
  2. Does the source support the direction, number, population, and certainty in the sentence?
  3. Does surrounding context change the interpretation, for example a limitation or a contrasting result?

If the passage does not answer those questions, find the original study or revise the sentence. A relevant title and a working link are useful starting points, not substitutes for reading the evidence.

Choose the sources before choosing the report

Deep Research is useful when a question calls for a broad survey across the web. Current versions can also use uploaded files and connected sources. For a comparison of the workflows, see ChatGPT Deep Research and Agent Bayes.

A library of papers serves a different purpose. When a supervisor has assigned a reading list, or a project depends on a collection of PDFs, you can select the papers first and ask questions within that collection. The answer is limited by the papers you included, so add relevant work as you find it.

In Agent Bayes, a mindmap is a workspace for organizing claims and their evidence. Search uses the project's libraries, and citations connect claims to passages in the papers. AI Verification can assess a claim against evidence in those libraries and explain when the evidence only partly supports it. A verification result is another assessment to inspect alongside the cited passage.

Try a claim with your own papers

Create a project in Agent Bayes, add papers to a library, and attach the library to the project. Ask the research agent to compare a finding across those papers, then open a citation and read the passage in context. Indexing and AI work use credits, with the charge based on the work performed.

For more on why source support matters, read how citation support verification works. For the distinction between ChatGPT's research modes, see our guide to agent mode and deep research.

New posts, straight to your inbox

No newsletter fluff, just an email when we publish something new.

Email me when a new post is published on the Agent Bayes blog. You can unsubscribe anytime. We'll first send a confirmation email, and we only use your details for this. See our Privacy Policy.

Enjoyed this? Share it.

GZ

Written by

Guy Zana · Founder

We are researchers and engineers building tools that help people reason over large bodies of literature without losing the thread back to the source.