- Scalene
- Posts
- Scalene 34: Assistance / PAIRR / Expanding
Scalene 34: Assistance / PAIRR / Expanding
Chris Leonard
April 13, 2025
Humans | AI | Peer review. The triangle is changing.
There are a couple of innovations this week from the world of education where the concept of peer reviewing is somewhat different in practice (students for students on classroom essays) but very similar in concept to academic research peer review. Plus, are we ready for chatbot-assisted review, and another upvote for constructive, positive-leading reviews…
13th April 2025
1//
The assisted Technology dilemma: a reflection on AI chatbots use and risks while reshaping the peer review process in scientific research
AI & Society - 17 March 2025 - 29 min read
I’m not sure how this ended up in my Scalene folder, but thank you to whoever sent me this paywalled article that I wouldn’t be able to access otherwise. This review of the current state of AI and peer review is a touch uncritical and could do with editing - for instance, the authors spend a few hundred words rehashing a blog post from Haseeb Irfanullah (more on him below) - and AI is referred to as ‘ai’ in the subheadings. However, the concluding call to action is on the money:
We need to wisely embrace the benefits of technology in improving academic practices while being aware of its limitations and potential dangers. We need to find a wise and fair balance between technological progress and maintaining rigorous, ethical academic standards (Mollick 2024). Moreover, ongoing research into the impact of AI on PR quality and outcomes is crucial for informed decision- making and policy development. The future of academic research and publication depends on our ability to manage this equilibrium, with a firm dedication to comprehensive, expert-driven PR. We call upon the academic community to collaborate in maintaining and evolving these standards, ensuring that our scholarly discourse not only preserves its integrity and quality but also embraces innovation responsibly, setting a new paradigm for ethical and effective PRPs in the age of AI.
2//
Level Up Peer Review in Education: Investigating genAI-driven Gamification system and its influence on Peer Feedback Effectiveness
arXiv.org - 03 April 2025 - 45 min read
This paper introduces Socratique, a gamified peer-assessment platform integrated with Generative AI (GenAI) assistance, designed to develop students’ peer-review skills in a functional programming course. By incorporating game elements, Socratique aims to motivate students to provide more feedback, while the GenAI assistant offers real-time support in crafting high quality, constructive comments.
CL: Give this a read, whilst imaging how useful this could be for academic journals. Not all AI-assistance necessarily focuses on replacing humans, some focuses on improving the quality of human review.
3//
Peer and AI Review + Reflection (PAIRR): A Human-Centered Approach to Formative Assessment
SSRN - 22 Feb 2025 - 21 min read
The PAIRR model connects AI feedback to peer review, a long-standing best practice in writing pedagogy. Student writers’ interactions with others about their writing correlates with deeper learning, as well as personal and social development. Moreover, many studies demonstrate that students learn from providing criteria-based feedback. Because the quality of peer review depends on how well it is scaffolded, AI feedback may be an especially valuable complement to peer review in courses across the curriculum, where instructors and TAs commonly lack training in writing pedagogy and where students receive little or no formative feedback. PAIRR emphasizes reflection and self-assessment and encourages students to be active and critically engaged users of AI feedback, rather than passive recipients who merely follow directions. In order to improve as writers, students need to assess exemplar texts, their own writing, their processes, and the feedback they receive.
4//
Peer Review Has Lost Its Human Face. So, What’s Next?
The Scholarly Kitchen - 09 Apr 2025 - 7 min read
Haseeb Irfanullah has pulled the pin and thrown another hand grenade into the Scholarly Kitchen, with predictable results. I have to confess to being a big fan of Haseeb’s writing and viewpoints, and his seminal Scholarly Kitchen post on a gradual transition to AI-based peer review is something which made me want to start this newsletter and give a more positive viewpoint to how AI can disrupt this bottleneck in publishing.
Here he posits 7 problems with peer review in 2025 and two potential solutions - which you can judge for yourself how feasible they may be. I think the professionalisation of peer reviewing (8.2 in his blog post) is a great idea, but maybe not within the authors’ institutions. Join the conversation in the comments - some fascinating insights in there too:
https://scholarlykitchen.sspnet.org/2025/04/09/peer-review-has-lost-its-human-face-so-whats-next/
5//
Peer review is meant to prevent scientific misconduct. But it has its own problems
The Conversation - 21 Mar 2025 - 4 min read
Another look at peer review problems, but with an uplifting ending. Also, I hadn’t seen the Channel 4 news coverage of The Rat paper before and it’s worth clicking through for that alone - but another way to improve peer review is suggested:
A “strength-based approach” to review can be encouraged. This is where feedback about the paper’s strengths as well as the gaps in the paper makes the feedback more “developmental” and less focused on what’s wrong with the paper.
From my experience as a journal editor, authors also find it helpful to receive the reviewers’ comments together with an overall summary from the editor highlighting the key issues raised by the reviewers.
And finally…
A paper I really wanted to feature this week, but I’m refusing to pay $47 for, is linked below. I’ve seen a few posts and press releases about it, but can only see the first page. However, if you have access, I think it will be worth your while:
The Challenges and Future of Peer Review - Critical Care Medicine
Also kudos to the APA who are mentoring peer reviewers in this news from April 1st. I hope it isn’t a joke!: https://www.apa.org/monitor/2025/04-05/scholarly-publishing-peer-review
Free consultation calls
Many of you may know I work for Cactus Communications in my day job, and one of my responsibilities there is to help publishers speed up their peer review processes. Usually this is in the form of 100% human peer review, delivered in 7 days. However, we are keen to experiment further with subtle AI assistance. If you want to chat about how to bring review times down with either a 100% human service, or you’re interested in experimenting with how AI can assist, let’s talk: https://calendly.com/chrisle1972/chris-leonard-cactus
Curated by me, Chris Leonard.
If you want to get in touch, please simply reply to this email.