• Scalene
  • Posts
  • Scalene 22: Preprints again, friend/foe, M3DocRAG

Scalene 22: Preprints again, friend/foe, M3DocRAG

Humans | AI | Peer review. The triangle is changing.

Here’s a surprise for you - a midweek Scalene. Due to being away this last weekend I was unable to send one on Sunday - but there’s just too much good stuff out there to wait until next weekend, so here we are. Come and talk to me about any of this at STM London conference.

26th November 2024

// 1
The Obsolescence of Traditional Peer Review: Why AI Should Replace Human Validation in Scientific Research
Preprints.org - 05 Nov 2024 - 10 min read

This article presents a radical reassessment of scientific validation processes, arguing that traditional peer review has become an outdated, inefficient, and ultimately flawed mechanism for ensuring research quality. Modern artificial intelligence systems demonstrate superior capabilities in analyzing methodological rigor, statistical validity, and literature comprehensiveness, while being free from human cognitive biases, professional rivalries, and institutional politics. Through examination of empirical evidence, we demonstrate how AI systems consistently outperform human reviewers in speed, accuracy, and comprehensiveness of research evaluation. The current peer review system, characterized by months-long delays, substantial costs, and demonstrable biases, actively impedes scientific progress. We propose a fully automated AI-driven validation framework that can evaluate research in real-time, identify methodological flaws, verify statistical analyses, and assess significance within the broader scientific context. This transformation would democratize research validation, eliminate publication bottlenecks, and accelerate scientific progress while maintaining higher standards of methodological rigor than currently possible under human review.

CL: I’ve included the full abstract here as it is so well written I didn’t want to butcher it with clumsy editing. This goes a step further than I would right now, but I might have the same opinion in 6-12 months.

[UPDATE 01 DEC 2024: IT IS WITH REGRET AND A LITTLE EMBARRASSMENT THAT I FLAG UP TO READERS THAT THIS PREPRINT IS COMPROMISED IN SEVERAL SERIOUS WAYS, BUT MAINLY BY BUILDING AN ARGUMENT ON MADE-UP REFERENCES. I THEREFORE RETRACT MY RECOMMENDATION THAT YOU READ IT.]

// 2
Preprints Discovery
Preprints.io- 22 Nov 2024 - 4 min read

From their blurb, Preprints Discovery is a means to track “potentially influential arXiv CS preprints using advanced bibliometric analysis to identify impactful research, early.” Sounds good. And their advanced bibliometrics analysis is described on the site. Not a replacement for peer review, but a potential stepping stone to removing human dependencies on deciding likely long-term value of papers.

We analyze the citation network formed by a preprint's references, treating it as a Directed Acyclic graph. Our algorithms evaluate network centrality, diversity, and temporal relevance to gauge the paper's theoretical foundation and potential influence. This helps identify work that builds meaningfully on established research while pushing boundaries into new territories.

Research quality often correlates with author expertise and track record. We aggregate comprehensive metrics for all co-authors, including h-indices, citation patterns, and publication history. This multi-dimensional analysis helps predict the potential impact and reliability of new research, even before it accumulates its own citations. Our final score combines these metrics using a weighted algorithm that has been calibrated against historical data, helping identify research that goes on to make significant impact in its field.

// 3
AI and Peer review: friends or foes

Two similarly-titled papers came to my attention on the same day and show some softening to the earlier stance publishers took when it came to incorporating AI into peer review. A hybrid solution is touted as being the best way forward here:

AI and Peer Review: Enemies or Allies? - Inside Higher Ed - 24 Oct 2023

// 4
Evaluating science: A comparison of human and AI reviewers
Cambridge CORE - 21 Nov 2024 -24 min read

We sought to understand (i) whether AI versus human reviewers are able to distinguish between made-up AI-generated and human-written conference abstracts reporting on actual research, and (ii) how the quality assessments by AI versus human reviewers of the reported research correspond to each other. We conducted a large-scale field experiment during a medium-sized scientific conference, relying on 305 human-written and 20 AI-written abstracts that were reviewed either by AI or 217 human reviewers. The results show that human reviewers and GPTZero were better in discerning (AI vs. human) authorship than GPT-4. Regarding quality assessments, there was rather low agreement between both human–human and human–AI reviewer pairs, but AI reviewers were more aligned with human reviewers in classifying the very best abstracts. This indicates that AI could become a prescreening tool for scientific abstracts.

CL: AI already is a pre-screening tool for abstracts. Contact me if you want details.

// 5
M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding
arXiv - 07 Nov 2024 - 28 min read

M3DocRAG, a new framework for answering questions from multi-page documents. It overcomes challenges faced by existing methods by using a multi-modal approach that includes visual elements and accommodates different document contexts. The authors also present a new benchmark, M3DocVQA, to evaluate this framework's effectiveness on over 3,000 PDF documents.

And finally…

Free consultation calls
Many of you may know I work for Cactus Communications in my day job, and one of my responsibilities there is to help publishers speed up their peer review processes. Usually this is in the form of 100% human peer review, delivered in 7 days. However, we are keen to experiment further with subtle AI assistance. If you want to chat about how to bring review times down with either a 100% human service, or you’re interested in experimenting with how AI can assist, let’s talk: https://calendly.com/chrisle1972/chris-leonard-cactus

Let's do coffee!
I’m in London for the STM meeting on December 4th (± 1 day).

Curated by Chris Leonard.
If you want to get in touch with me, please simply reply to this email.