Surprisingly, current AI reviewers are competitive even with the top-rated reviewers in Nature’s official peer review: on the composite of correctness, significance, and evidence sufficiency, they produce a significantly higher fraction of review items per paper than the top-rated human reviewer. AI reviewers also show distinctive strengths: thorough cross-reference checking, code-level inspection, and raising valid criticisms that human reviewers miss. The weaknesses are equally clear: AI reviewers produce more factually incorrect items than humans do, with the composite advantage arising from higher significance and evidence sufficiency among items that are correct. AI reviewers’ items also overlap with each other far more than human reviewers’ do, so an all-AI panel would substantially narrow the diversity of perspectives. Finally, AI reviewers exhibit characteristic failure patterns rarely seen in humans: a limited grasp of subfield-specific methodological conventions, losing track of content across long papers and supplementary materials, and an overly critical stance that inflates minor issues.

To conduct the case study, we build a web UI-integrated tool, AI-Paper-Review, that generates structured AI review of a draft paper, available at https://github.com/unarylab/ai-paper-review. This tool selects several AI reviewers from a diverse pool of AI reviewers and clusters and ranks their comments based on commonality and importance of review comments. It also allows to align AI comments with human comments to facilitate metric-based validation. The case study shows that AI review can cover a significant fraction of human-raised issues, but also raises issues missing in human review.

Our analysis reveals that the generated reviews diverge significantly from feedback provided by human reviewers: LLM ratings are less variable, positively biased, and overconfident, and their cross-reference patterns are model-dependent and distinct from human norms. Furthermore, when evaluated through PRAIB, we observe that LLMs tend to generate longer, more complex reviews, yet frequently overlook the atomic weaknesses noted by human reviewers. By characterizing where and how LLMs reviewing behavior departs from human norms, PRAIB provides the community with a diagnostic tool for identifying which aspects of the review process LLMs can reliably support today and which require further development before deployment.

LLMs are currently unsuitable as wholesale replacements for human expertise. Instead, we propose a synergistic pipeline: leveraging frontier models for mathematical analysis and fine-tuned models for stylistic drafting, while retaining human reviewers for final qualitative judgment and citation verification.

At intake (target: hours), a stakes router classifies the manuscript and routes it accordingly. In parallel, four agents extract methods, scan for reproducibility against code and data, build a structured claim graph, and match candidate reviewers using both topic expertise and methodological fit. The output is a structured submission packet that an editor can read in minutes rather than days.

At editorial orchestration (target: days), a human editor reads the packet, sees the stakes classification, and decides routing. Acceptance for review, request for pre-review revision, or rejection, each requires a human signature, and the editor's queue is itself transparent.

At review (target: weeks, in parallel rather than serial), human peer reviewers focus on what humans do uniquely well, novelty, significance, contextual judgment, while AI co-reviewers handle methodological rigor, statistical claims, prior-art conflicts, and reproducibility checks. Both feed into a shared structured rubric.

The output is not "accepted" or "rejected." It is a living trust profile attached to the document, a versioned text with multi-dimensional badges, a visible stakes tier, and continuous inputs flowing in afterward: replications, contradicting evidence, author updates, verified comments.

In this paper, we establish that current AI systems fail the necessary conditions for peer review automation. However, meeting these conditions would not automatically justify full automation. Questions of accountability, democratic legitimacy, and measurement validity require explicit community deliberation. We call for a rigorous science of peer review automation to address such questions. We envision transparent science that empirically evaluates tools before deployment, studies how humans interact with AI assistance, and develops incentive structures that maximize the value of human expertise.