- Scalene
- Posts
- Scalene 57: Necessary / Growing / Reflection
Scalene 57: Necessary / Growing / Reflection
Chris Leonard
April 04, 2026
Humans | AI | Peer review. The triangle is changing.
Following up on my promise of 2 in a week, here’s the second instalment of all things peer review and AI. Alongside this newsletter I’m also working on a presentation for the SSP meeting next month which is going to focus on the concept of personal peer review lenses. Stay tuned for more on that.
4th April 2026
1//
AI-assisted Reviewing is Necessary and Should be Open
Open AI Review - Mar 2026 - 10 min read
I came across Chenhao Tan’s blog post via a LinkedIn post promoting OpenAIReview - an automated peer review platform which performs in a similar way to many other automated review platforms - but with one difference - this is open.
He first posits that AI review is necessary, then lays out his arguments for making things open: to promote equity, customisation, iteration, improvement, and honesty - and they are eloquently elucidated herein. The reports from the web interface are decent and, given the open nature of the code, can be refined and customised.
2//
AI’s Growing Role as Scientific Peer Reviewer
Stanford University - 26 Mar 2026 - 4 min read
A brief interview with James Zou, who has been active in researching the effectiveness of AI peer review, but also was one of the organisers of the recent ‘Agents for Science’ conference where contributions were both written and reviewed by AI agents.
…as AI becomes a routine scientific collaborator — writer, coder, critic — the scientific community will have to keep refining which roles belong to machines, which belong to people, and how to make this relationship both useful and trustworthy.
3//
Ninety-seven ignored: A personal reflection on the hidden struggles of an academic editor
Eur. Sci. Editing - 09 Mar 2026 - 05 min read
It’s great to remind myself occasionally about why we need to define the terms around how we introduce AI to the peer review process. It’s because of stories like this. Himel Mondal describes the difficult and emotionally draining job of being an academic journal editor in the 21st century:
Over the course of handling around 20 articles, I found the task increasingly difficult. The true challenge was finding reviewers. On average, I needed to send requests to at least 30 reviewers to get two acceptances. For one article, I sent out over 100 reviewer invitations. None accepted. Not a single reviewer.
For other manuscripts, those who did accept often delayed their responses by months. And when I finally received a review, it was sometimes so brief that the technical team rejected it and insisted I find more reviewers.
4//
882 Review Requests. One Researcher. One Year.
Maxim Topaz - 28 Mar 2026 - 1 min read
And it’s not just editors who are struggling. Topaz, a professor at Columbia University, has created a visual breakdown of just what a burden peer review could be to him if he assented to every invitation he received. If editors and reviewers are struggling, how are we going to resolve this situation?
5//
Preprints.ai: How Much of Peer Review Can We Automate?
Openresearch.wtf - 23 Mar 2026 - 9 min read
When Mark Hahnel has something to say, it’s usually worth listening. Here he’s looking at peer reviewing preprints - one of the obvious steps in disintermediating journals from the scholarly communications process.
Preprints.ai asks a sharper question than "can AI help with peer review?" It asks “if we were designing peer review from scratch for a world where powerful LLMs exist, what would we actually need humans for, and what could we comfortably automate?”
The platform currently runs a multi-agent review pipeline against preprints from bioRxiv and medRxiv, assessing two dimensions: research integrity (methodology, statistical validity, reproducibility, citation accuracy) and novelty (does the core claim already exist in the literature?). The output is a structured assessment graded on an A5-to-E1 rubric, with positivity bias actively recalibrated using publication outcome data.
And finally…
Sabir Foux pits LLMs against each other to reach consensus on peer review outcomes: https://www.linkedin.com/pulse/ai-peer-review-why-i-never-take-single-model-face-value-sabir-foux-2mxpe/
Reviewer3.com benchmarked 80,000+ human and AI review comments. Here’s what they found: https://reviewer3.com/evidence/reviewbench
Policies Permitting LLM Use for Polishing Peer Reviews Are Currently Not Enforceable: https://arxiv.org/abs/2603.20450
That’s it - we’re up to date! See you next time for a ‘normal’ update.
Let’s chat
I’m always happy to chat - just reply to this email.
Curated by me, Chris Leonard.
If you want to get in touch, please simply reply to this email.