This is the final Scalene of 2024. I’ll be spending my Sundays in holiday mode for the next few weeks and hope to be back and have something sitting in your inbox on 5th January. Honestly, that feels like a long time away at the moment with the promising Gemini 2 and ChatGPT Pro just coming out, but they’ll just have to wait. We’re going to round out 2024 by looking at a bumper crop of stories for you to digest with your mulled wine.

15 December 2024

// 1
Results of the NeurIPS 2024 Experiment on the Usefulness of LLMs as an Author Checklist Assistant for Scientific Papers
NeurIPS Blog - 11 Dec 2024 - 6 min read

It’s great to get some quantitative and qualitative insight on the use of LLMs in assessing academic research from a cohort of people who know what they’re talking about. Enter the chairs of the Neural Information Processing Systems conference, who ran an experiment with their conference where authors could opt in to an automated evaluation of their submission to see if it complied with submission standards - a fairly low-risk experiment.

What was eye-opening for me, was how enthusiastic the submitters were and how they were excited to see what feedback they got - and then the subsequent reality check of it not living up to their expectations:

The reasons for their unhappiness largely stemmed from the LLM being inaccurate, or too strict. There is also a fascinating discussion into whether and how this automated system could be gamed (it can, and therefore cannot replace human judgement right now).

A summary blog post is here: https://blog.neurips.cc/2024/12/10/results-of-the-neurips-2024-experiment-on-the-usefulness-of-llms-as-an-author-checklist-assistant-for-scientific-papers/
The full paper describing the results are here: https://arxiv.org/pdf/2411.03417

// 2
Near future academic publishing – a speculative social science fiction experiment
Learning, Media and Technology- 08 Dec 2024 - 8 min read

Rather than simply look forward to 2025 in this editorial, the authors extrapolate current shifts in academic publishing to 2030 and 2035. As with all these thought experiments, while I agree with some of the predictions, I think they will all happen much sooner than the authors expect. There is a doomsday scenario predicted for the near future, but a positively Utopian longer-term vision outlined for 2035 (see 2030 and 2035 quotes below).

https://doi.org/10.1080/17439884.2024.2436835

// 3
An Oath of Research Integrity?
AJ Boston - 06 Dec 2024 - 3 min read

I love Arthur’s writing for its style, and there is always something of substance to make it doubly enjoyable. Here he articulates the solution I can see having the most pragmatic likelihood of success in an AI-mediated scholarly communication landscape.

Make the authors (and I would add, reviewers) take an oath, like a doctor’s Hippocratic oath, where they publicly declare they will adhere to the best accepted standards in research.

A stronger and publicy-visible declaration of responsibility for your own output may be the only sensible way to absorb AI assistance in research. Sure, use LLMs for your peer review, but you sign your name on it and be prepared to stand behind any recommendations therein.

https://aj-boston.pubpub.org/pub/integrity-oath/release/10

// 4
“Does it feel like a scientific paper?”: A qualitative analysis of preprint servers’ moderation and quality assurance processes
MetaArXiv - 18 Sept 2024 -16 min read

This is an analysis of the ‘acceptance’ criteria for various preprint servers as told by the moderators for those services. As you may know I believe preprints + automated reviews will become an important channel of communication in the near future, so I’m intrigued to see what factors are considered important for a ‘desk reject’ at the moment. The tables at the end of the document are particularly rich in value for prospective authors.

In conjunction with another piece of work published recently [Mehmani & Malički’s Structured peer review: implementation and checklist development] there are a wealth of granular prompting suggestions someone could use to come up with a very useful tool for authors.

You can read them both here:
https://doi.org/10.31222/osf.io/mp6ky
https://doi.org/10.3897/ese.2024.e137675

// 5
WithdrarXiv (+SciFy)
arXiv - 10 Nov 2024 - 17 min read

I love this, a dataset of 14,000 manuscripts withdrawn from arXiv, annotated with their reasons for withdrawal. The authors describe the development of a taxonomy of retraction reasons, identifying 10 categories, and a simple yet accurate automatic classification of the same.

But the paper becomes more interesting when discussing WithdrarXiv-SciFy

I won’t steal their thunder by listing them here, you can click on the link below to read them in section 7. But needless to say it represents an opportunity to enable some automated verification of scientific claims and mathematical theorem proving, as well as discrepancies between data and figures/text.

https://arxiv.org/abs/2412.03775

And finally…

Lots and lots of ‘other’ stuff here. here. Why not click on them all to open in a new tab and read them when time permits over the holidays?

• Confronting the demonization of AI writing: Reevaluating its role in upholding scientific integrity

• https://x.com/alek_dimi/status/1868097488458273214/?rw_tt_thread=True

• In Praise of Peer Review

• ‘Getting paid to review is justice’: journal pays peer reviewers in cryptocurrency

• Trust in AI: progress, challenges, and future directions

• https://scholarlycritic.com - I’m not sure the website (which looks like a predatory journal) is helping its mission here, but this is a site focused on poor peer review and publishing practices in business research. Interesting nonetheless.

And I couldn’t let the festive period pass without linking to this. Merry Christmas to you all.

Curated by me, Chris Leonard.
If you want to get in touch, please simply reply to this email.