Back to normal this week, although with multiple sports-related distractions. I’m excited to share what’s new in the space, so let’s go…

14th July 2024

// 1
Assessing ChatGPT's ability to emulate human reviewers in scientific research: A descriptive and qualitative approach
sciencedirect.com - 01 July 2024 - 5 min read (summary)

https://doi.org/10.1016/j.cmpb.2024.108313

CL - I love this experimental design which compares 3 human reviewer opinions with that of a ChatGPT4-generated report for manuscripts submitted to a certain journal. As expected, there are strong areas of overlap, and other areas where ChatGPT4 agrees less with the human reviewers. I’d like to see this repeated with 4o and Claude 3.5, but right now my takeaway is that a hybrid approach is likely to lead to optimal reports, combining things that humans missed, and using humans to weed out irrelevant or incorrect assertions from the LLM.

// 2
Insights 2024: Attitudes toward AI
Elsevier - July 2024 - 55 min read

Elsevier have been on a bit of a roll recently with timely thought leadership pieces encompassing the opinions of researchers, university administrators, and funders. Their latest examination of attitudes to AI in research is–as its name suggests–insightful, although I was a little confused as to some of the answers around AI and peer review.

A previous study (Research Futures 2.0) suggested that 21% of researchers would read an article which had undergone exclusively AI peer review as they expect it to be more free of bias and consistent across all submitted manuscripts - but 59% strongly disagreed, valuing human understanding above any AI process.

Now, however, it seems 93% of respondents believe that AI will bring benefits to publication process in the authoring, reviewing, and impact monitoring aspects of the work. Attitudes seem to be changing, all be it with the sensible caveat that AI’s influence on a review report should be noted.

Link to full report below. There are some excellent links at the bottom of the page (Databooks) which delve into the data more deeply and granularly (if that’s a word?).
https://www.elsevier.com/en-gb/insights/attitudes-toward-ai

// 3
PEER: Expertizing Domain-Specific Tasks with a Multi-Agent Framework and Tuning Methods
arXiv.org - 10 July 2024 - 24 min read

CL: I love the use of multiple agents here to determine how to answer the question and then decide if the answer is good enough. If not, it goes through another iteration.
https://arxiv.org/abs/2407.06985v2

// 4
Viewpoint: the evolving landscape of peer review
Emerald.com - 12 March 2024 - 22 min read

This review was recommended to me recently and I finally got round to reading it this week. It’s a great examination of the current state of peer review more broadly, but struggles with answering it’s own questions on ChatGPT as a reviewer for academic articles.

https://doi.org/10.1108/JSM-09-2023-0325

// 5
Structured peer review: pilot results from 23 Elsevier journals
PeerJ - 25 June 2024 - 20 min read

I’ve been delaying commenting/posting on this study for my own reasons, but it should be of interest to anyone who wants to speed up and unbias peer review.

https://peerj.com/articles/17514/

CL: So what is my hesitation here? Nothing to do with the study, which is a solid piece of work, but rather with the idea of homogenising peer review reports. It’s my belief that peer review will become the discerning service between journals in the future, and journals who don’t offer something different to their rivals, risk losing submissions. If your peer review service and report was better and different than your rival, that’s something that will attract more submissions, no? Maybe mid-tier journals within a publisher portfolio would benefit from this for cascade purposes, but not the top-level journals.

And finally…
So, England made it to the final of Euro2024, and it’s the topic of conversation amongst most people on the streets right now. But I’m not going to jinx anything by mentioning anything about it. Instead I’m sharing this image, which is something I have to fight when writing this newsletter too!

Curated by Chris Leonard.
If you want to get in touch with me, please simply reply to this email.