The escalating volume of academic research, coupled with a shortage of qualified reviewers, necessitates innovative approaches to peer review. While large language model (LLMs) offer potential for automating this process, their current limitations include superficial critiques, hallucinations, and a lack of actionable insights. This research addresses these challenges by introducing a comprehensive evaluation framework for AI-generated reviews, that measures alignment with human evaluations, verifies factual accuracy, assesses analytical depth, and identifies actionable insights. We also propose a novel alignment mechanism that tailors LLM-generated reviews to the unique evaluation priorities of individual conferences and journals.

Peer review processes are slowing. Existing literature and policies conceptualize this stagnation in peer review as a result of academic fatigue. Here, we instead examine an under-researched factor behind slowed peer review systems: academics refusing to voluntarily review manuscripts for for-profit journals. By synthesizing accounts of peer review refusal from scholarly blogs, journal editorials, and prominent social media movements, we provide an overview of the advantages and disadvantages of this refusal. We then offer some preliminary suggestions for academics to simultaneously safeguard the speed of peer review and voice dissatisfaction with major publishing companies.

Peer review is the foundation on which the scientific world is built. However, it’s a flawed process, as proved by a number of controversial publications. Is it time for change? ‘We don’t even know if it works at all.’

This review aims to evaluate the perspectives of the scientific community on integrating AI tools in the peer review process, focusing on their applications, limitations, and ethical considerations. This study seeks to bridge the gap between the technological advancements offered by AI and the foundational principles of academic publishing, such as transparency, integrity, and objectivity. By identifying best practices, the review aims to provide actionable and evidence-based recommendations for effectively incorporating AI into peer review workflows to maintain the quality and fairness of the process.