• Scalene
  • Posts
  • Scalene 46: Real work / radiology / FAIR

Scalene 46: Real work / radiology / FAIR

Author

Chris Leonard
October 05, 2025

Humans | AI | Peer review. The triangle is changing.

I’m writing this week’s newsletter from the departure lounge at Heathrow Terminal 3 - hence no images and a somewhat streamlined approach. And amidst this international hubbub, I’m wondering how to cover everything I wanted to cover in a single newsletter, and I’ve come to the conclusion that I can’t - or rather, there is a better way to handle this surfeit of stories. I’ve siphoned off all the arXiv links this week and will present them in a separate email later this week. In fact I might eat some of my own dog food and have a go at AI-mediated evaluation, as well as curation. But I promise I am not turning into a journal!

5th October 2025

1//
AI Peer

Website - March 2025

AI Peer is a group of researchers based in Sheffield and Oslo who have the vision to “investigate the extent to which large language models and other artificial intelligence methods can support peer and expert review of academic documents. This includes developing and evaluating algorithms and strategies.”

They have been active as a group since March 2025 and have already come up with some evidence-backed assertions that may raise eyebrows (Finding 1 and 2 at the link below) and look like they have their eyes set on raising some more over the coming months. Definitely worth looking at their plans and future output.

2//
Real AI Agents and Real Work

oneUsefulThing - 29 Sept 2025 - 6 min read

Ethan Mollick casts his eye over the replication crisis and posits that AI agents are well placed to replace humans (where there are humans doing this) in the very near future. He explains how Claude 4.5 was able to verify a tricky economics paper with the assistance of agents - breaking down complex problems into simpler, smaller ones, and running analyses in parallel before collating results and correcting for errors where appropriate. Now imagine running this as part of the review process, rather than retrospectively.

But the revolutionary part is not that I saved a lot of time. It is that a crisis that has shaken entire academic fields could be partially resolved with reproduction, but doing so required painstaking and expensive human effort that was impossible to do at scale. Now it appears that AI could check many published papers, reproducing results, with implications for all of scientific research. There are still barriers to doing this, including benchmarking for accuracy and fairness, but it is now a real possibility. Reproducing research may be an AI task, not a job, but it is also might change an entire field of human endeavor dramatically. What makes this possible? AI agents have gotten much better, very quickly.

3//
Biomedical publishing: Past historic, present continuous, future conditional

PLOS Biol - 03 Oct 2023 - 33 min read

That’s not a typo above, this is a 2-year old paper. Positively ancient by my usual standards, but something which has been taking up a lot of headspace for me in the last few weeks. Richard Sever predicts the future of academic publishing, and seems to have gotten the broad strokes right. I love this excerpt from the abstract:

The Web created the potential for a more decoupled publishing system in which articles are initially disseminated by preprint servers and then undergo evaluation elsewhere. To build this future, we must first understand the roles journals currently play and consider what types of content screening and review are necessary and for which papers. A new, open ecosystem involving preprint servers, journals, independent content-vetting initiatives, and curation services could provide more multidimensional signals for papers and avoid the current conflation of trust, quality, and impact.

The whole paper is an easy read if you’re into this stuff. I recommend taking the time to read it, but also reflect on where we are headed. For the record, I believe that preprint servers, automated assessment, and overlay (virtual) journals are going to be massively more important in the very near future.

4//
AI isn't replacing radiologists

Works In Progress - 25 Sept 2025 - 14 min read

It’s hard to believe this is only the second time I’ve recommended something from my favourite online writing outlet - Works in Progress. They go deep on subjects, do their own research, and present everything in an easy to read, beautiful-looking website.

One of my favourite studies about AI is the one where AI, humans, and then AI+humans are asked to identify anomalies in radiography scans. AI got 98% right, humans 96%, and AI+humans 93% (presumably due to humans overruling correct AI decisions). And I’m damned if I can find the link to that right now. Apologies, treat the data points as indicative rather than absolute for now.

Anyway, the point is that AI - for all its accuracy in radiology - isn’t replacing radiologists. Some of the reasons for that are mundane, but some are unusual. I won’t spoil the pleasure of reading this by revealing too much - but what are the corollaries with peer review here?

5//
The FAIR framework: ethical hybrid peer review

J. Perinatal Med. - 23 Sept 2025 - 17 min read

Saving the best til last - this is an amazingly thorough and well-presented framework for peer review in the AI era. Kudos the Journal of Perinatal Medicine for publishing this.

And finally…

It’s been a while since we’ve had anything funny here, so thanks to Shahan Ali Memon for the smiles this week:

The future of peer review is here. Introducing Nature Instant! We've automated rejection so you can get disappointed faster. Because why wait 6 months for reviewer 2 when you can get roasted by an #AI in seconds?

I mean, it’s funny, but it’s also not when you spend a few minutes thinking about it. As Shahan points out, this is less speculative than it was when he designed it 2 years ago.

https://bsky.app/profile/shahanmemon.bsky.social/post/3m2gh6ng5ds2g
[read the whole thread!]

Oh - and I nearly forgot the OUP Peer Reviewer survey. I will be returning to this in depth in a future issue. Read all about it here:
https://academic.oup.com/pages/journal-peer-review-survey-report?ref=the-geyser.com

Let’s chat

I’ll be at the STM Frankfurt and Frankfurt Book Fair in mid-October. Wanna meet?

Curated by me, Chris Leonard.
If you want to get in touch, please simply reply to this email.