Welcome to issue 11. Stories on fraud and reviewer mills highlight that the bar for AI-assisted review really isn’t as high as some people think. There’s a great-looking new book highlighted, and I find another gem from arXiv from way back in the mists of 2023. I’m taking a summer break now, so see you all again in a few weeks.

4th August 2024

// 1
Can large language models provide useful feedback on research papers? A large-scale empirical analysis
arXiv.org - 03 Oct 2023 - 32 min read

Over 3,000 accepted papers and nearly 9,000 open peer reviews from the Nature portfolio were used as a basis in this study, which even though it’s a nearly a year old now (gasp!) is still a brilliant and even-handed analysis of specifically how and where LLMs (GPT4 in this case) can add value to the review process. Specifically they found that LLM feedback can be vague, but can spot things human reviewers miss. LLM feedback suffers from context window fatigue, i.e. it concentrates more on the first part of the paper than the rest of it, and that LLM feedback doesn’t ‘sound’ human.
They recommend that it be used as a tool by authors, and by human reviewers, but not as a replacement for human review. All of this still holds true in my eyes even for currently available systems. A great foundational paper.
https://arxiv.org/abs/2310.01783

// 2
Scientific Publishing Ecosystem, a partial book review
Springer Nature - 3 Aug 2024 - 55 min read

It’s not often I do (free!) marketing for multinational corporations, but I have been super impressed by a new book from SN looking at all components of the academic publishing system. Take a look at the TOC here.

I can’t remember when I last put my hand in my own pocket to buy an academic ebook, but then again I can’t remember when just the Table of Contents excited me so much. So I did it. And since it was only out yesterday, I haven’t read it all, and won’t pretend I have, but I did skip to the chapter entitled Scientific Peer Review in an Era of Artificial Intelligence. It’s a very comprehensive look at research integrity as well as traditional peer review, and looks forward as well as back, although I was a little underwhelmed at section 23.5.1 - a few hundred words on Integrating AI into Peer Review feels like a missed opportunity, but maybe, like all blockbusters, there is a sequel to come soon? There are no references from 2024, which is part of the nature of book publishing I’m afraid. Anyway, a solid 4/5 from me:

https://link.springer.com/chapter/10.1007/978-981-97-4060-4_23

// 3
The Academic Culture of Fraud
palladiummag.com - 02 Aug 2024 - 15 min read

This is a tough read - and not just because of subheadings such as ‘The Epistemic Bankruptcy of Academia’. Although it is exceedingly well-written, and the author’s displeasure at the subject feels almost visceral, it’s a tough read for anyone involved in peer review. And not just at high-volume OA publishers; Nature comes in for the worst criticism. It’s tough because it speaks a truth that perhaps has been unspoken - some of us knew or had suspicions of publications with impaired peer review processes. The work to undo that damage takes too long, and does not have sufficient repercussions to stop people trying it again and again, at enormous cost to science and peoples’ lives.
https://www.palladiummag.com/2024/08/02/the-academic-culture-of-fraud/

Hated that? You may also be upset by:
The Peer Review Crisis - insidehighered.com - 12 June 2022
The Hindawi Files - Part 1 - James Heathers - 20 July 2024

// 4
Systems thinking and peer review
ce-strategy.com - 01 Aug 2024 - 11 min read

Taking a cue from systems thinking and micro services architecture, Clarke & Esposito’s Colleen Scollans and Pam Harley investigate how journal submission and peer review systems could benefit from a modular, interconnected set of services, allowing for flexible and customisable workflow solutions (after all, every journal is different).
https://www.ce-strategy.com/2024/08/revolutionizing-manuscript-submission-and-peer-review-systems-through-systems-thinking/

// 5
Review mills are the next horror to undermine academic publishing
Springer Nature - 01 August 2024 - 24 min read

Now, I’m a great admirer of how much good stuff Springer have been publishing recently, but even I’m not paying for 2 things from them in the same week, so thanks to a free trial of DeepDyve (not an ad) I was able to read this paper, and quickly wished I hadn’t.
Somehow a small cabal of reviewers are providing the same plagiarised, vague and unrelated comments to many papers, and coupling that with coercive citation demands for their own papers, making them look like highly-cited, community-minded scientists who do lot of reviewing work, thereby getting more review requests.
Most of the data discussed in the paper is available - for free - as a supplementary excel file, where you can link to the reviews and the relevant dissections of them on PubPeer. It seems cruel to highlight that these are almost all MDPI papers, but at least they were being transparent with sharing the reviews in the first place:

https://doi.org/10.1007/s11192-024-05125-w

This story reminds me of something Ethan Mollick said in a recent blog post. AI doesn’t have to be perfect, it needs to be better than the best available human. And these available humans have set the bar quite low.

And finally…
This is the last Scalene newsletter in August. I’m going away camping in France for a few weeks and, although we’ll miss each other, it will be all the sweeter when I return with issue 12 on August 31st. A bientôt!

Curated by Chris Leonard.
If you want to get in touch with me, please simply reply to this email.