How AI summaries erase scholarly disagreement
A balanced-sounding synthesis can be wrong about the shape of a debate. Why fluent summarization flattens contested literature, a taxonomy of the disagreements that get lost, and a writing method that keeps them visible.

Ask a language model to summarize three papers that disagree and you will usually get a paragraph smoother than the literature deserves. It will be readable. It will sound balanced. And it can be wrong about the one thing a literature review exists to capture: the shape of the debate.
For a researcher, disagreement is not noise to be averaged out of the signal. It is often the signal. Which camps exist, what evidence each one rests on, whether they are arguing about findings or about definitions, and what would settle it: that structure is the actual product of a literature review. A summary that dissolves it into "studies suggest mixed results" has not summarized the field. It has replaced the field with something easier to write.
This post is about how that erasure happens, a taxonomy of the disagreements that get lost, and the method we use to keep them visible, which works in a mindmap, a document, or a plain text file.
A coherent paragraph can be an inaccurate map of a field
Consider a contested question with three sources behind it. Paper A reports that an intervention works, in a specific population, with a modest effect. Paper B finds no effect in a broader population. Paper C argues that A and B are measuring different constructs under the same name, so their results cannot be compared at all.
A conventional synthesis prompt, "summarize what these papers say about X," reliably produces something like: "Research on X shows promising though mixed results, with effectiveness varying across populations and measurement approaches."
Every clause of that sentence is defensible. The paragraph as a whole is a distortion. It renders C's methodological objection, which if correct dissolves the entire "mixed results" framing, as a gentle qualifier. It implies A and B are points on one scale when C's whole argument is that they are not. A reader walks away believing the field has a maturing consensus with some noise, when the field actually has a live argument about whether its central construct is coherent.
The failure is not inaccuracy at the sentence level. It is misrepresentation at the structure level, and it is invisible unless you already know the literature, which is precisely when you do not need the summary.
Why does this happen? A summarization objective rewards a single fluent answer, and instructions like "provide a balanced synthesis" push the model toward a central position phrased confidently. Whether the mechanism is in the training or the prompt, the observable output pattern is the same: contested fields come back smoother than they are.
Contradiction is only one kind of disagreement
Part of the problem is that "the papers disagree" covers at least six different situations, and they deserve different treatment:
- Conflicting findings. Two studies measure the same thing and get incompatible results. The clean case, and the rarest.
- Competing explanations. The observations are compatible, the causal stories are not. Both camps can cite the same data.
- Methodological disagreement. One paper argues another's design cannot support its conclusion. This is an attack on evidence quality, not a rival finding.
- Scope differences. One study is about adolescents, another about adults. Not a contradiction at all until someone overgeneralizes, at which point it becomes one.
- Definitional disagreement. The camps use one term for different constructs. The debate is upstream of any finding.
- Differences in certainty. One author asserts, another suspects. Same direction, different epistemic weight.
A flattening summary destroys the distinctions between these. Worse, a crude "preserve disagreement" instruction can mangle them in the opposite direction, rendering a scope difference as a dramatic conflict. Not every difference between papers is an opposition, and forcing differences into pro and con camps is its own distortion, sometimes called false balance. The taxonomy matters because the goal is fidelity to the kind of disagreement, not the manufacture of debate.
Attribution errors turn a literature review into academic telephone
There is a second, sneakier way synthesis erases the structure of a field: getting who-said-what wrong.
Academic prose is full of positions the author does not hold. A paper opens by stating the consensus it intends to demolish. A review reports a primary study's finding without endorsing it. An author assumes a premise for the sake of argument, or presents a hypothesis explicitly as untested. Retrieval finds these passages, and they are genuinely relevant. But if the synthesis step reads them flat, the demolished consensus becomes the author's view, and the review's neutral report becomes its own finding.
We have watched this happen in our own system, and wrote up a real case: while mapping the imposter syndrome literature, the agent attributed a striking statistic to a review paper when the review was only summarizing someone else's primary study. The full story is in our post on that mapping exercise. The citation was real, the page was real, and the attribution was still wrong. One miscitation like that, quoted onward by the next reviewer, is how a field ends up with claims that everyone cites and no one can locate.
This is why retrieval relevance is not enough. The system, or the researcher, has to read what the passage is doing rhetorically, not just what it says.
Represent positions as structure before writing prose
The method that defends against all of this is old, and it does not require any particular tool: represent the disagreement as explicit structure before anyone writes connected prose.
Contested question: does X improve Y?
├── Position: X improves Y (modest effect)
│ ├── Evidence: Paper A, RCT, population P1
│ └── Scope: effect shown only for P1
├── Position: X shows no effect
│ ├── Evidence: Paper B, cohort study, population P2
│ └── Scope: broader population, different outcome measure
├── Objection: A and B measure different constructs
│ └── Evidence: Paper C, measurement analysis
└── Unresolved: no study uses both measures on one population
The tree is not the literature review. It is the inspectable intermediate from which a faithful review gets written. Its virtue is that it makes erasure visible: you cannot silently average two branches, and an empty "unresolved" node stares at you until you either fill it or admit the gap is real.
From that structure, faithful prose follows a method you can apply mechanically:
- State the question the sources actually diverge on.
- Attribute every position to its holder, by name.
- Keep the scope, population, and method attached to each finding.
- Preserve each author's level of certainty, including hedges.
- Say what evidence would distinguish the accounts, and leave unresolved what is unresolved.
The result is less quotable than a confident paragraph. It is also what a literature review is for.
What this costs, honestly
Preserving structure is not free, and pretending otherwise would undersell the judgment involved.
A map of a contested field is harder to read than a summary of it, and for a genuinely settled question, a unified synthesis is the correct output; representing a fringe objection as a co-equal branch misleads in the other direction. Deciding which differences deserve separate positions, and which are variations within one, is interpretive work. No rule replaces it. The structure does not remove the researcher's judgment. It gives the judgment something concrete to operate on, instead of asking it to reverse-engineer a finished paragraph.
How we encode this in a research agent
We built Agent Bayes around exactly this intermediate representation: a persistent mindmap where claims are nodes, every cited node points to the passage and page behind it, and the editing agent operates under written rules that come straight from the failures above. Conflicting views must become sibling nodes, never one averaged claim. A view an author introduces as a foil must be represented as a reported view, and misattributing it is classified in the instructions as a citation fidelity error, the same severity as a wrong page. Epistemic qualifiers survive on purpose: a conjecture stays a conjecture.
The rules are architecture, not aspiration, but they are also not magic. The agent can still misjudge a rhetorical role, which is why every node stays pinned to its passage: the misjudgment is one click from being caught, which is the property that actually matters.
Audit one paragraph of your own review
You can test all of this on your own writing in about five minutes, no tools required. Take one synthesis paragraph from your current literature review. Underline every claim and write down who, specifically, holds it. Circle every hedge in the original sources and check whether it survived into your prose. Then ask the uncomfortable question: does the paragraph assert a consensus that none of the underlying papers actually states?
If it does, you have found the exact spot where a fluent sentence swallowed a debate. That is the spot worth rewriting, whether or not an AI wrote the first draft.
New posts, straight to your inbox
No newsletter fluff, just an email when we publish something new.
Email me when a new post is published on the Agent Bayes blog. You can unsubscribe anytime. We'll first send a confirmation email, and we only use your details for this. See our Privacy Policy.