• Scalene
  • Posts
  • Scalene 10: Paper SEA / Boston Research Journals / Claude as reviewer

Scalene 10: Paper SEA / Boston Research Journals / Claude as reviewer

Author

Chris Leonard
July 28, 2024

Humans | AI | Peer review. The triangle is changing.

Welcome to issue 10. This week we’re featuring an overlooked gem from arXiv, a journal proudly explaining how they use AI in peer review, and a very direct use of Claude in reviewing social science papers. Let’s dive in shall we?

28th July 2024

// 1
Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and
arXiv.org - 09 July 2024 - 43 min read

Although existing methods have explored the capabilities of Large Lan- guage Models (LLMs) for automated scientific reviewing, their generated contents are often generic or partial. To address the issues above, we introduce an automated paper reviewing framework SEA. It comprises of three mod- ules: Standardization, Evaluation, and Anal- ysis, which are represented by models SEA- S, SEA-E, and SEA-A, respectively. Initially, SEA-S distills data standardization capabilities of GPT-4 for integrating multiple reviews for a paper. Then, SEA-E utilizes standardized data for fine-tuning, enabling it to generate con- structive reviews. Finally, SEA-A introduces a new evaluation metric called mismatch score to assess the consistency between paper con- tents and reviews. Moreover, we design a self- correction strategy to enhance the consistency.

https://arxiv.org/abs/2407.12857

CL - I should have caught this a couple of weeks ago, but I’m glad it didn’t slip through the net completely. It’s particularly worth looking into the supplemental information in the appendix. Lots of practical examples which show the evolution of the report generation from pithy and general to detailed and focused. They used various Mistral models and ChatGPT, so the obvious next step is to use Claude (see story 5 in this newsletter), as well as refining it for use in other scientific domains.

// 2
The ethics of using artificial intelligence in scientific research: new guidance needed for a new tool
AI & Ethics - 27 May 2024 - 65 min read

This is a long, but worthwhile read on the ethics of using AI in research. It comes up with nine recommendations for the responsible use of AI, ranging from my favourite - that researchers are responsible for identifying, describing, reducing, and controlling AI-related biases and random errors in their own work - to the more controversial use of synthetic data. It comes with in-depth descriptions of the fundamental technologies which power these AI systems too. Maybe one for the sun lounger when you’re on holiday…

// 3
New RFI – Recommendations on the Use of AI in Scholarly Communication
EASE - 23 July 2024 - 3 min read

This toolkit aims to promote responsible and transparent use of AI by editors, authors, and reviewers, with links to examples of current policies and practices. As AIs are fast evolving the PRC will be monitoring and updating these recommendations as new information becomes available. Please contact them to share any opinions, policies, and examples that could help them improve this guide.

EASE Members can discuss the document in the Forum, under this post created by PRC Chair, Mario Malicki.  Why not read the document and add any questions you may have to the forum before completing the form?

The Peer Review Committee looks forward to engaging in conversation on AI and learning what works and does not work in the proposed steps.

CL: Peer review section is a strong focus of this and EASE members can comment until 15th September (just before Peer Review Week 2024):
https://ease.org.uk/2024/07/new-rfi-recommendations-on-the-use-of-ai-in-scholarly-communication/

// 4
Boston Research Journals
BRJ - 27 July 2024 - 6 min read

A publisher which has sailed under my radar is Boston Research Journals. However I was very interested to read about their peer review process, and the fact that proudly explain how AI is helping reduce the time this takes. I am unaware of many other journals who promote this on their own websites, but BRJ do. Briefly they use AI to find the best reviewers, do two double blind reviews, and then generate an AI report which looks at technical and content issues. The handling editor then has three reports with which to make a final decision. Hybrid reviewing is the (near) future:
https://bostonresearch.org/author-guidelines/

// 5
Claude 3.5 is a solid reviewer of social science papers.
LinkedIn - 27 July 2024 - 3 min read

…not my words, but those of Ethan Mollick on LinkedIn. He continues:

Just did an experiment where I reviewed a paper on my own, then, after I was done, had Claude do the same. We overlapped on 70% - I found key issues it did not, but it also found points I missed. Useful as second opinion.

Prompt (Try it with your own papers): This paper was submitted to the X Journal. You are a critical reviewer offering detailed and helpful feedback to help the authors. First, read through the paper and list the major issues you see that need help. then rate it on the following scales [pasted in criteria given to reviewers]. Finally give me a detailed review letter.

It was pretty harsh to one of my papers actually (but not wrong)

CL: Claude is certainly one of the best models to turn out what look like good peer review reports, but it still throws up errors, false negatives, and generally (but not exclusively) provides a blinkered report focused on the manuscript itself and not linking it to the work that has preceded it. However, at the authoring/presubmission stage this is a great author tool. For definitive peer review, we still need humans to verify and expand on AI output. Hybrid reviewing is the (near) future. Click the link to read the report it generated:

And finally…
Lots of ‘other’ things I’ve been reading this week, but since I value your time (and attention), I don’t intend to extend this newsletter beyond 5 highlights each week. However, I am providing and recommending these links without further commentary:

Let's do coffee!
I’m travelling to the following places over the next few weeks. Always happy to meet and discuss anything related to this newsletter. Just reply to this email and we can set something up:

Birmingham: 29th July
Leeds: 8th August
Oxford 9th August
ALPSP (Manchester): 11-13 September
Frankfurt Book Fair: 14-18 October

Curated by Chris Leonard.
If you want to get in touch with me, please simply reply to this email.