We introduce AbsenceBench to assesses LLMs' capacity to detect missing information across three domains: numerical sequences, poetry, and GitHub pull requests. AbsenceBench asks models to identify which pieces of a document were deliberately removed, given access to both the original and edited contexts. Despite the apparent straightforwardness of these tasks, our experiments reveal that even state-of-the-art models like Claude-3.7-Sonnet achieve only 69.6% F1-score with a modest average context length of 5K tokens. Our analysis suggests this poor performance stems from a fundamental limitation: Transformer attention mechanisms cannot easily attend to "gaps" in documents since these absences don't correspond to any specific keys that can be attended to.

If LLMs can’t tell what’s missing, this might pose challenges for using LLMs as judges, graders, or assistants in any domain where absence matters. We hope AbsenceBench can serve as a starting point for building more robust and trustworthy architectures that are absence-aware: it’s a necessary but far from sufficient bar for making sure LLMs can reason about absence.

This study evaluates the effectiveness of varying levels of human and artificial intelligence (AI) integration in reproducibility assessments of quantitative social science research. We computationally reproduced quantitative results from pub- lished articles in the social sciences with 288 researchers, randomly assigned to 103 teams across three groups — human-only teams, AI-assisted teams and teams whose task was to minimally guide an AI to conduct reproducibility checks (the “AI-led” approach). Findings reveal that when working independently, human teams matched the reproducibility success rates of teams using AI assistance, while both groups substantially outperformed AI-led approaches (with human teams achieving 57 percentage points higher success rates than AI-led teams, p < 0.001).

🤖⚖️ “Our AI is 90% accurate!”
“Great, let’s add a human in the loop for the final 10%.”
"Perfect, now it’s 78%!"
…. beyond ~80% model accuracy, adding human-in-the-loop mechanisms actually degrades overall system performance. Not because people are malicious or lazy. We bring inconsistency, and often override good recommendations for the wrong reasons.

In this article we argue that crowd-sourced peer review is likely to do better than journal-solicited peer review at sorting articles by quality. Our argument rests on two key claims. First, crowd- sourced peer review will lead on average to more reviewers per article than journal-solicited peer review. Second, due to the wisdom of the crowds, more reviewers will tend to make better judgements than fewer reviewers will.

Scientists should probably be putting out shorter narratives, datasets, code, and models at a faster rate, with more visibility into their thinking, mistakes, and methods. In this age of the internet, almost anything could technically be a “publishable unit.” It doesn’t even have to sound nice or match the human attention span anymore, given our increasing reliance on AI agents.

In more general terms, we need publishing to be a reliable record that approximates the true scientific process as closely as possible, both in content and in time. The way we publish now is so far from this goal that we’re even preventing our own ability to develop useful large language models (LLMs) that accelerate science. Automated AI agents can’t learn scientific reasoning based on a journal record that presents only the glossy parts of the intellectual process, and not actual human reasoning. We are shooting ourselves in the foot.