“If we can automate evaluations”, says Thelwall, then “it would be a massive productivity boost”. And there’s potential for huge savings: the most recent REF, in 2021, was estimated to have cost around £471 million (US$618 million).

One prominent essay forcibly arguing against peer review is “The Rise and Fall of Peer Review,” by Adam Mastroianni. In it, Mastroianni calls peer review a “failed experiment” and concludes that the best path forward is to do away with it altogether. He suggests that peer review doesn’t “do the thing it’s supposed to do” — namely, “catch bad research and prevent it from being published,” so perhaps scientists would be better off uploading PDFs to the internet where review and oversight can happen post-publication…

…We have all heard stories of high-quality work getting rejected while mediocre or error-laden work slips through (numerous high-profile fraud cases come to mind). It’s also undeniable that we lack enough data to say whether we get sufficient returns from peer review relative to the enormous amounts of time, resources, and energy we invest in it. Overall, issues such as these are why I found — and continue to find — many of Mastroianni’s arguments compelling….

…Still, shortly after I read Mastroianni’s article, Ben Recht asked on Twitter if there were any articles presenting “a positive case for peer review.” And while many individuals responded to Mastroianni, and a smattering of defenses of peer review have been published, I found it difficult to locate a comprehensive steelman of the practice. This article attempts to fill that lacuna.

At some point in the future, it seems likely that we will run out of reviewers. As to what happens then, well your guess is as good as mine. The old chestnut of paying reviewers as an extra incentive will no doubt come up again. After all, most journals are published by for-profit companies, and I cannot think of another example where highly qualified professionals are asked to donate their time, completely unpaid, in a situation where a third party makes a significant financial profit from their efforts.

We talk about the use of AI in Peer Review as well as how academic culture can, and should, change so that reviewers are better recognized for the work they do for journals and publishers. We also get Zeger and Martin's feedback on the recently filed US Class Action Suit that deals with peer review.....

Papers are submitted by the authors to scientific venues, and these papers must be reviewed by peers or other authors. The meta-reviewers then gather the peer reviews, assess them, and create a meta-review and decision for each manuscript. As the number of papers submitted to these venues has grown in recent years, it becomes increasingly challenging for meta-reviewers to collect these peer evaluations on time while still maintaining the quality that is the primary goal of meta-review creation. In this paper, we address two peer review aggregation challenges a meta-reviewer faces: paper acceptance decision- making and meta-review generation. Firstly, we propose to automate the process of acceptance decision prediction by applying traditional machine learning algorithms. We use pre-trained word embedding techniques BERT to process the reviews written in natural language text. For the meta- review generation, we propose a transfer learning model based on the T5 model. Experimental results show that BERT is more effective than the other word embedding techniques, and the recommendation score is an important feature for the acceptance decision prediction. In addition, we figure out that fine-tuned T5 outperforms other inference models. Our proposed system takes peer reviews and other relevant features as input to produce a meta-review and make a judgment on whether or not the paper should be accepted