• Scalene
  • Posts
  • Scalene 32: Detection / Support / Bridging

Scalene 32: Detection / Support / Bridging

Author

Chris Leonard
March 26, 2025

Humans | AI | Peer review. The triangle is changing.

Busy time in professional and personal life at the moment means I haven’t had the opportunity to update on all things Scalene recently, but we’re here now, so let’s get on with it!

26th March 2025

1//
Detecting LLM-Written Peer Reviews

arXiv.org - 20 March 2025 - 53 min read

Existing tools for detecting LLM-generated content are not designed to differentiate between fully LLM-generated reviews and those merely polished by an LLM. In this work, the authors employ a straightforward approach to identify LLM-generated reviews — doing an indirect prompt injection via the paper PDF to ask the LLM to embed a watermark. Their tests show a high success rate in identifying LLM reviews with no false positives.

2//
Understanding and Supporting Peer Review Using AI-reframed Positive Summary

arXiv.org - 13 March 2025 - 59 min read

While peer review enhances writing and research quality, harsh feedback can frustrate and demotivate authors. Hence, it is essential to explore how critiques should be delivered to motivate authors and enable them to keep iterating their work. In this study, we explored the impact of appending an automatically generated positive summary to the peer reviews of a writing task, alongside varying levels of overall evaluations (high vs. low), on authors’ feedback reception, revision outcomes, and motivation to revise. Through a 2x2 online experiment with 137 participants, we found that adding an AI-reframed positive summary to otherwise harsh feedback increased authors’ critique acceptance, whereas low overall evaluations of their work led to increased revision efforts. We discuss the implications of using AI in peer feedback, focusing on how AI-driven critiques can influence critique acceptance and support research communities in fostering productive and friendly peer feedback practices.

CL: Peer review in AI doesn’t necessarily mean it replaces human judgement. In this instance it enhances the experience for the author without impacting the perceived expertise of the reviewer.

3//
Bridging Social Psychology and LLM Reasoning: Conflict-Aware Meta-Review Generation via Cognitive Alignment

arXiv.org - 21 Mar 2025 - 46 min read

The Cognitive Alignment Framework (CAF) leverages dual-process cognitive modeling to enhance meta-review synthesis in LLM-based systems and eliminate different kids of biases. By incorporating a three-phase processing pipeline, CAF effectively manages conflicting viewpoints, enabling more balanced and coherent meta-reviews. The experimental results validates the effectiveness of the proposed CAF across multiple LLM-based models. This framework features a three-phase process that enhances fairness and consistency by addressing issues like the anchoring effect and conformity bias. Anyone familiar with Daniel Kahneman’s Thinking Fast And Slow will recognise elements of this paper.

//4
Early Career Researchers on all Aspects of Peer Review: A Deep Dive Into the Data

Learned Publishing - 17 Feb 2025 - 41 min read

A study of early career researchers found that most are experienced in peer review and generally trust the process, though they have concerns about its effectiveness. Many believe peer review could be improved, especially with the introduction of AI, which they see as potentially beneficial. Researchers in Portugal and Malaysia were most vocal in support of AI replacing humans altogether at some point. Page 11 onward for the AI part of this survey.

https://doi.org/10.1002/leap.2002open_in_new

//5
The potential contribution of generative AI to journal peer review

Sex, Drugs, Economics - 17 Mar 2025 - 6 mins

Pataranutaporn et al.'s paper is focused on solving a "peer review crisis". It has become increasingly difficult to find peer reviewers who are willing to spend the time to generate a high-quality review that will in turn help to improve the quality of published research. Generative AI could help to alleviate this, but we're clearly not entirely there yet. There is still an important role for humans in the peer review process, at least for now.

If we’ve learned anything over these last 31 newsletters, it’s this. I wonder when we’ll be justified in having the optimism of Portuguese and Malaysian ECRs?

And finally…

I’ll be keynoting (is that a verb?) at ALPSP Redux in Oxford next week. The brief is to examine how AI will affect university presses and will be much more wide-ranging than just peer review, so come and join us on April 3rd at The Mathematical Institute. Also, now the dust has settled on the Sakana AI-generated paper story, TechCrunch came up with some interesting takes on it below:

Let's do coffee!
- ALPSP UP Redux, Oxford - April 3-4

Free consultation calls
Many of you may know I work for Cactus Communications in my day job, and one of my responsibilities there is to help publishers speed up their peer review processes. Usually this is in the form of 100% human peer review, delivered in 7 days. However, we are keen to experiment further with subtle AI assistance. If you want to chat about how to bring review times down with either a 100% human service, or you’re interested in experimenting with how AI can assist, let’s talk: https://calendly.com/chrisle1972/chris-leonard-cactus

Curated by me, Chris Leonard.
If you want to get in touch, please simply reply to this email.