A few examples this week of how great prompting can result in great outputs from LLMs. It really can be worth your while crafting very descriptive prompts if you want to get the most out current AI tools, especially as it relates to peer review. I’m also now offering free 15-minute consultations to anyone who is interested in speeding up their peer review workflows. See link at bottom for more details.

17th November 2024

// 1
The Art of Prompting in Peer Review
LinkedIn - 14 Nov 2024 - 2 min read

I’m always fascinated by people on LinkedIn like Ethan Mollick who are continually showing us how to use AI tools in smart ways, and demonstrating the limits of what they can do. A lot of it is to do with the art of prompting. I had watched a long YouTube video this week on How to Win with Prompt Engineering, when I then came across this great example on LinkedIn.

The author, Kenny Unice, used the Prompt-Query Alignment Model (PQAM) structured prompt principles to get a fantastic result. Look at the effort and thought that went into that prompt, and then consider that once it’s done, you can use that prompt over and over again.

Before the holiday period, I’m going to be sharing some of my own tactics around automated manuscript evaluation using NotebookLM and Claude in particular. Stay tuned!

// 2
Can AI review the scientific literature — and figure out what it all means?
Nature- 13 Nov 2024 - 10 min read

This is an examination of how well LLMs can generate review articles. While not exactly about peer review, the problems and suggested solutions are the same. It’s not specific enough, they hallucinate, RAG systems are better but not perfect, and the overall outcome is “at the level of an undergraduate student who pulls an all-nighter and comes up with the main points of a few papers”. But it was this quote that stuck out for me:

My guess is both of those estimates are way off. We seem to be ignoring the exponential nature of improvements within the quantum steps we take with each new generation of LLM. If I were a betting man, I’d be erring on the side of 2-3 years. https://www.nature.com/articles/d41586-024-03676-9

// 3
When using Artificial Intelligence Tools in Scientific Publications Authors should include the Prompts and the Generated Text as Part of the Submission
Journal of Academic Ethics - 13 Nov 2024 - 1 min read

It’s hard to argue with any of this. When I’m looking at preprints on arXiv, the best parts are the appendices and supplementary files which show exactly how the researchers got their results. There is no reason not to do this for all articles.

Weirdly this paper is not open access, but I think the title and abstract give a good flavour of what is there for non-subscribers.

https://doi.org/10.1007/s10805-024-09581-0

// 4
Preprints at a crossroads – Are we compromising openness for credibility?
LSE Impact blog - 13 Nov 2024 -7 min read

I love any discussion about the future of preprint servers as I am convinced they are going to play a large part in the dissemination of academic research in an age of (largely) automated review. We may think preprints have had their day in the sun, but I’m of the belief we have only just started with them. So this opinion piece was surprising to me as it ended with these words:

Maybe the value of virtually-instantaneous review reports to be published alongside the preprint manuscript hasn’t been considered here, but it’s fairly obvious to me that preprint servers could be one of the first adopters of automated review tools (caveat lector, etc.)
https://blogs.lse.ac.uk/impactofsocialsciences/2024/11/13/preprints-at-a-crossroads-are-we-compromising-openness-for-credibility/

// 5
CycleResearcher: Improving Automated Research via Automated Review
arXiv - 28 Oct 2024 - 34 min read

Another gem on arXiv, this time looking at a wider issue of using open-source (LLMs) to automate the entire research process, including literature review and peer review. It introduces two new datasets and - of interest to us - shows that the CycleReviewer model outperforms human reviewers in predicting paper scores, much like the MAMORX paper we highlighted last week. It’s only a matter of time isn’t it?!
https://arxiv.org/abs/2411.00816

And finally…

Some other great reads that you may appreciate:

• What happened to the pursuit of truth? - J Gen Physiol - 23 Oct 2024

• Rethinking Reviewer Fatigue - Eon - 15 Nov 2024

• The impact of artificial intelligence on scholars: an interview with Juan D. Machin-Mastromatteo - Digital Library Perspectives - 29 Oct 2024

X is an odd place right now. It feels like a pub where a gang of rowdy young men have come in, talking loudly and aggressively and you know it’s not going to get any better (in fact, only worse), but you don’t really want to leave. This is another account I’m going to miss when I’ve eventually left my seat in the X pub - Mascot’s Minute Silence. It’s hard to pay respects to anyone when you’re dressed as a smiley swan or laughing shrimp, as these pictures illustrate.

Possibly no newsletter next week as I’m away in Yorkshire, but we’ll see.

Curated by Chris Leonard.
If you want to get in touch with me, please simply reply to this email.