- Scalene
- Posts
- Scalene 48: AI+DEI / BadScientist / Positive review only
Scalene 48: AI+DEI / BadScientist / Positive review only
Chris Leonard
November 09, 2025
Humans | AI | Peer review. The triangle is changing.
Too much stuff to share this time. I’m not going to waffle on, let’s just get into it. A few arXiv papers again, so I’ve shared a Claude-generated peer review so you can judge the worthiness of these preprints yourself.
9th November 2025
1//
AI in peer review: can artificial intelligence be an ally in reducing gender and geographical gaps in peer review? A randomized trial
Research Integrity Journal - 27 October 2025 - 14 min read
A fascinating paper looking at how adding a single line to a GPT-4o prompt can radically improve the range of suggested peer reviewers that are returned. Honestly, I’d never considered using a LLM to find reviewers before, but if you are doing that, you probably need to read this and add in the relevant prompt.
Without DEI, GPT-4o mostly suggested male scientists from high-income countries.
With a DEI prompt, the suggested pool became gender balanced and more geographically diverse. Academic metrics (publications, citations, h-index) were similar across both conditions.
2//
“Give a Positive Review Only”: An Early Investigation Into In-Paper Prompt Injection Attacks and Defenses for AI Reviewers
arXiv - 03 Nov 2025 - 14 min read
I’ve had a bit of experience with prompt injections in peer review, and while I remain skeptical of the success rate of these, one thing is clear, authors are trying it in a variety of ways. I’ve always been of the opinion that pdfs should be re-rendered by OCR/computer vision tools to avoid these hidden instructions, but that isn’t always going to happen.
Authors will say this is to protect themselves from lazy reviewers. If that were the case, the hidden prompts could be somewhat more neutral rather than urging the LLM to give a positive review. I suggest that an email is triggered to the author alerting them to what has happened and letting the EIC decide on the validity of the resulting review.
This review looks at state-of-the-art models and prompt injections and concludes it is a real problem which needs real solutions. This reader, at least, thinks this is a solvable problem.
Claude 4.5 peer review with Extended Thinking and Research:
https://claude.ai/public/artifacts/3f928bd5-02ba-45c9-8c64-59620286a53c
3//
BadScientist: Can a Research Agent Write Convincing but Unsound Papers that Fool LLM Reviewers?
arXiv - 20 Oct 2023 - 33 min read
This paper was a bit of an eye-opener for me, introducing BadScientist, a framework that pits fabrication-oriented paper generation against multi-model LLM review systems. The five tools to manipulate LLM output into passable papers is particularly interesting and surely could be used in some more ethical sense.
Our findings expose a critical vulnerability: LLM review systems can be systematically deceived by presentation manipulation. Fabricated papers achieve high acceptance rates, with reviewers frequently exhibiting concern-acceptance conflicts—flagging integrity issues yet still recommending acceptance. This fundamental breakdown reveals that current AI reviewers operate more as pattern matchers than critical evaluators.
Our mitigation attempts show the inadequacy of current defenses. Detection accuracy barely exceeds random chance, and paradoxically, adding explicit integrity checks sometimes increases acceptance rates. Simply asking LLM reviewers to "be more careful" is insufficient.
The scientific community faces an urgent choice. Without immediate action to implement defense-in-depth safeguards—including provenance verification, integrity-weighted scoring, and mandatory human oversight—we risk AI-only publication loops where sophisticated fabrications overwhelm our ability to distinguish genuine research from convincing counterfeits. The integrity of scientific knowledge itself is at stake.
Claude 4.5 peer review with Research and Extended Thinking:
https://claude.ai/public/artifacts/96a45085-faa8-4ca5-8b86-bf5a82dbd104
arXiv blog - 31 Oct 2025 - 5 min read
A remarkable posting from the administrators of arXiv. Review articles and Position papers in Computer Science will now not be accepted to the world’s largest preprint server unless they have already been accepted by a journal or major conference. And the reason seems to be AI-generated slop:
In the past few years, arXiv has been flooded with papers. Generative AI / large language models have added to this flood by making papers – especially papers not introducing new research results – fast and easy to write. While categories across arXiv have all seen a major increase in submissions, it’s particularly pronounced in arXiv’s CS category.
This opens up so many questions, but should also be a canary in the coalmine for journals too. AI-generated papers are getting out of control, they need weeding out before review, and reviewers should be aware of the characteristics of AI-generated pieces in these fields.
5//
ReviewGuard: Enhancing Deficient Peer Review Detection via LLM-Driven Data Augmentation
arXiv - 18 Oct 2025 - 34 min read
ReviewGuard tackles deficient peer review detection through LLM-powered classification, training models on ICLR/NeurIPS reviews augmented with synthetic data. While addressing a pressing integrity concerns (both rising AI-generated and low-quality reviews) the approach faces significant limitations: training relies on abstracts rather than full manuscripts, limiting contextual understanding; validation remains confined to top ML/AI conferences, raising generalizability questions across disciplines; and the framework paradoxically employs GPT-4.1 to detect flaws potentially introduced by similar systems. Despite this, I think it still provides editors with preliminary quality-control tools.
Claude 4.5 review with Extended Thinking and Research:
https://claude.ai/public/artifacts/82361414-320f-4abd-b67e-ff61b37ae5e9
Weirdly, it has attributed this report to me personally.
1 year ago: Scalene 20 - 10 Nov 2024
Let’s chat
I’ll be at the STM London meeting on both days. Come and say hi.
Curated by me, Chris Leonard.
If you want to get in touch, please simply reply to this email.