• Scalene
  • Posts
  • Scalene 5: Multi-turns / Peer review quality / Enshittification of peer review

Scalene 5: Multi-turns / Peer review quality / Enshittification of peer review

Author

Chris Leonard
June 16, 2024

Humans | AI | Peer review. The triangle is changing.

Welcome to issue 5 of Scalene, examining how the interplay between humans, AI, and peer review is changing. I’m always keen to meet and talk to others with interests in this area, so I’m sharing my travel dates with you all. If you’re interested in coffee and a chat sometime, I’ll be in the following cities shortly: London - Thursday 20th June, Berlin - 26/27th June, Oxford - 10th July. Just reply to this mail and we’ll set something up. And now, on with the show…

16th June 2024

// 1
Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactions

In this paper, we reformulate the peer-review process as a multi-turn, long-context dialogue, incorporating distinct roles for authors, reviewers, and decision makers. We construct a comprehensive dataset containing over 26,841 papers with 92,017 reviews collected from multiple sources, including the top-tier conference and prestigious journal. This dataset is meticulously designed to facilitate the applications of LLMs for multi-turn dialogues, effectively simulating the complete peer-review process.

https://arxiv.org/abs/2406.05688v1

CL - One criticism of AI-assisted peer review is that it can only help in the generation of the peer review report. This approach takes a more holistic approach to the whole peer review process, including humans-in-the-loop for each stage. A fascinating read.

// 2
The advent of human-assisted peer review by AI

How peer review is carried out may be disrupted by AI. Today’s peer-review processes are asynchronous and often involve too many steps, they can be unnecessarily slow because of the substantial effort required and the need to get specialist expertise (peer reviewers are not always available), and they require the poring over a seemingly ever-increasing amount of text, figures, data and, increasingly, code. In the future, AI agents could be developed to carry out most of the publishers’ checks, help human editors assess the scientific context, perform peer review before human peer review, and relieve human experts from the drudgery of figuring out where the authors have placed a piece of data or what they meant to convey with a confusing schematic.

CL - This editorial in Nature Biomedical Engineering is refreshingly optimistic about the future of AI and peer review, all be it with some short-term caveats. I particularly like the hybrid human-AI approach to report generation: AI first, human second.
https://www.nature.com/articles/s41551-024-01228-0

// 3
How to Assess Peer Review Quality
In developing alternatives to human-dependent review, it’s often hard to know if what we’re doing is as-good or better than the existing solution. As anyone who has managed a journal peer review process will know, we tend to fetishise human peer review, despite it not being that great for a large part of the time. However, we need to hold ourselves to higher standards than a 2-month late, 3-sentence review. This is where these EASE guidelines come in super useful.
They involve rating the peer review by the author, the reviewer themselves, and the handling editor. They can even be run retrospectively to gain a baseline to measure any AI-assisted experiment against.
https://ease.org.uk/communities/peer-review-committee/peer-review-toolkit/how-to-assess-peer-review-quality/

// 4
Redefining the paradigm: incentivizing the peer review process for scientific advancement

Reviewers stand at a crossroads. They are torn between their commitment to contribute to the collective knowledge of their field and the allure of personal academic pursuits that promise greater professional advancement. In this light, peer review is increasingly perceived as a distraction, a detour from the path to recognition and career progression. This shift in perception has led to a gradual decline in enthusiasm for participating in the peer review process, as the potential rewards seem to pale compared to the effort and time invested..

// 5
T&F policy statement on AI for peer review
This week a new policy statement on the use of AI in the peer review workflow was issued by Taylor & Francis. I reproduce the relevant section here:

Peer reviewers are chosen experts in their fields and should not be using Generative AI for analysis or to summarise submitted articles or portions thereof in the creation of their reviews. As such, peer reviewers must not upload unpublished manuscripts or project proposals, including any associated files, images or information, into Generative AI tools.

Generative AI may only be utilised to assist with improving review language, but peer reviewers will at all times remain responsible for ensuring the accuracy and integrity of their reviews.

https://taylorandfrancis.com/our-policies/ai-policy/

CL - Nothing unexpected here by today’s standards, but I do implore publishers making these statements to include two pieces of information that would make them more valuable, namely a Published date and a ‘To Be Updated’ date. The field is moving quickly and this information is useful.

And finally…
I like to finish on a short read that is funny, uplifting, or pithy in some way. This examination of the state of peer review reminds us of why we need AI to speed up and unbias peer review:
Pee Review: The Enshittification of Science?:

PS: Many thanks to the Scalene reader who sent me a link to this paper I recommended blind last week. It’s not what I thought it would be, but a good read nonetheless. And now it gets a second mention!
A GAN-BERT based decision making approach in peer review
https://doi.org/10.1007/s13278-024-01269-y

Curated by Chris Leonard.
If you want to get in touch with me, please simply reply to this email.