Hi. This new newsletter aims to cover advances in peer review, largely but not exclusively, enabled by AI. It is my personal belief that AI will be able to write 'better-than-human' peer reviews within the next 5 years. What does that mean for the publishing industry, authors, and research as an endeavour, and how will we get there? At some point on this journey together, we'll find out.

A little bit about me (Chris Leonard): I've been working at the interface of editorial and technology within the scholarly comms industry for 25 years for the likes of Current Science, Elsevier, BioMed Central, SpringerNature, Bloomsbury/Qatar Foundation, Emerald, and now Cactus Communications . In that time I've seen a lot of change, but I haven't seen much change in one of the biggest bottlenecks in publishing, peer review.

Peer review is a relatively recent invention (that I can even say that something 100 years old is a 'relatively recent' invention speaks volume about the pace of change in STM publishing) and one which was clearly designed for different times. As we now generate 4m STM submitted manuscripts per year, requiring 8m reviews (as a minimum), you can see peer review is creaking at the seams. There are only an estimated 10m researchers worldwide.

And at the same time we are in the midst of a revolution. On the same scale at the worldwide web, the sudden and universal access to artificial intelligence (and specifically large language models) stands to upend many businesses and business models, and bring hitherto undreamed of improvements to many text-based operations.

Given that many publishers nowadays need to invite 20+ people to review a manuscript to get 2 acceptances, and then have to chase them up and then (all too frequently) get a substandard report back, shouldn't we be trying to see where AI can help here? We're still some way away from 100% AI-generated peer review reports that we can rely on, but AI can certainly aid reviewers today. Anything we can do to reduce review times from months to weeks to days should be the primary interest of most publishers today.

Objections to AI in peer review are frequently based on assumptions that the current exponential increase in capabilities will stop and not improve wildly every few months. It is my belief that peer review is about to brought into the 21st century and completely redefined for the current age. That's what this newsletter is about. I'm glad you're here for the ride.

I'll not be rambling on like this in every missive. This is just a scene-setter for the first email. And while we're scene-setting, why the name 'Scalene'? Well, there are shifting sands in the interplay between AI, humans and peer review processes, and how best to visualise that other than with triangles? At least it avoids use of the word 'axis' - which is something we should all be grateful for.

Oh and one more thing... I am employed by Cactus Communications. Although this is a personal newsletter, I will flag up conflicts of interest where appropriate. We are busy in this space too!

21st May 2024

IOPP: State of Peer Review 2024​
A bit slow getting on to this one (I've had a newsletter to set up, so bear with me), but the IOP Publishing report on the State of Peer Review 2024 is quite a read. Over 3,000 researchers responded to a survey set up by IOP asking about their experiences with peer review and specifically whether AI will help or harm the process.

Views were typically polarised. Someone with an eye for a soundbite suggested AI should be burnt with fire, whereas others were more optimistic. Very roughly it split opinion three ways - a third think it will improve peer review, a third think it will make it worse, and a third were neutral. Exact %s and other data can be found in the full report.

Another interesting tit-bit: researchers in Europe are receive the most requests for peer review, with 24% receiving three or more requests per month.

Length matters​
Despite what others may have told you, the length of a peer review does matter and to satisfy the author they should be at least 947 words long. That's a startling finding from a paper by Abdelghani Motti & Luis Miotti, handily summarised by the author in a post on the LSE​ blog:

I think we all know short reviews can be pithy and superficial, but the opposite case never really struck me for some reason. As an author, and editor, you would want something substantial - particularly if you'd been waiting a long time for it.

Paper-Wizard​
​And finally, I was pleasantly surprised to find this service a few days ago, all be it one that is shrouded in mystery. It appears, from the Ts&Cs, to originate from Australia - but no clues as to who is offering this service. It seems to be a fully AI-driven peer review report generator. However, with my one free credit I was not expecting what was returned to me. Briefly, if you ask a LLM to generate a peer review report (no matter how you prompt it) it almost always says minor/major reviews. Never straight accept or reject. Paper-Wizard (rightly) identified my paper as fundamentally flawed and gave a 6-page breakdown on what needs to change before it can be resubmitted elsewhere. Impressive.

You can try it yourself here. After your free review, it's $6 per manuscript, or $29 for 10 credits. Obviously there are caveats everywhere here, but it may represent a little taste of the future.

^{Scalene is published once a week. Usually on a Sunday, unless I’m doing something else.}