Issue 4 already! Where does the time go? It’s been another week of fascinating advances in the journey towards ending 100% dependence on humans for the evaluation of academic research, so let’s take a look at them. On a personal note, I will be in Berlin on June 27th for a great-looking meeting organised by Silverchair. I hope to meet some of you there.

9th June 2024

// 1
Automated Focused Feedback Generation for Scientific Writing Assistance

https://arxiv.org/abs/2405.20477

CL - This is exciting for a number of reasons, not least of which is that this approach builds a framework to evaluate the content of a manuscript, not just metadata and clarity of presentation. Specific, actionable feedback to improve the content of a paper at a granular level is the holy grail in this space.

// 2
The Other Reviewer: RoboReviewer
I came across a special issue of Journal of the Association for Information Systems recently, dedicated to Generative AI and Knowledge Work. After an ‘interesting’ user journey to find the issue in question, I think I found the full article list as a PDF.
All of the articles will catch your eye in one way or another, but the one that caught my eye the most was the one introducing the concept of RoboReviewer. The opinion piece by Ron Weber does a great job of looking at what peer review does, how we might start to use AI in the process, and introduces the concept of a ‘RoboReviewer marketplace’.
However it was Section 6 that shone for me. How will AI(-assisted) peer review change journal submission dynamics:

CL - Should tools like ‘RoboReviewer’ be available to authors? My gut says yes, but my head is conflicted. We don’t make image manipulation or research integrity tools easily available to authors for good reason. But this might improve the overall quality of submissions to journals too. Food for thought.
https://aisel.aisnet.org/cgi/viewcontent.cgi?article=2171&context=jais

// 3
Transforming the Editorial and Peer Review Process
I promised a review of outcomes from SSP, but there has been relatively little to report in terms of AI and peer review. However, this review of the STM conference from Highwire’s Tony Alves has some great insights. He describes how an AI-positive editor views the future of journal publishing, and how industry stalwarts (sorry Anita & Todd!) see us moving to this future state.
Specifically on how the world may look in 2034, the journal editor made this point:

https://www.highwirepress.com/blog/stm-us-annual-conference-notes-part-3-transforming-the-editorial-and-peer-review-process/

CL - most of these things are already options for journal editors and publishers - but if you are not one of those, contact me at my day job 😀 

// 4
The challenges of using AI tools in peer reviewing and the need to go beyond publishing policies
Thank you to the kind (and anonymous) reader who pointed me in the direction of this paper. After my observation last week that COPE don’t seem to have guidelines for AI use in peer review, this article also adds WAME to the list, but does provide a helpful overview of all publisher policies, where available:
https://journals.sagepub.com/doi/10.1177/17470161231224552

// 5
What can’t (easily) be replicated
This article from February, which already seems dated in referring to ChatGPT 3.5, does however make a good point which can get lost in occasional technological fervour:

https://pubmed.ncbi.nlm.nih.gov/38330745/

CL - Yes, but… this isn’t a task for the peer review report to concern itself with (whether that be generated human, AI, or a mix). Editors-in-Chief and publishers are still going to be required to make these judgements for a long time into the future. I’ll dedicate a future newsletter to wider issues like this. If you can’t wait that long, take a look at this piece in The Yuan, which will scratch an itch for you.

And finally…
This short opinion piece in ACM Communications by Daniel Lemire covers two related topics: the “diarrhoea” of LLM-generated papers, and the ability of AI to evaluate research. He argues we should focus on outcomes (impact), not outputs (papers) and that we could be on the cusp of a Golden Age for scientific progress. A little dose of optimism to start your week:
https://cacm.acm.org/blogcacm/artificial-intelligence-is-the-crisis-we-need/

PS: I wanted to review this paper, but have no access to it. Based on the title and abstract, I’m going to go out on a limb and recommend it anyway. Maybe you can read it:
A GAN-BERT based decision making approach in peer review
https://doi.org/10.1007/s13278-024-01269-y

Curated by Chris Leonard.
If you want to get in touch with me, please simply reply to this email.