AI-Driven Reproducibility and Analytical Robustness in Scientific Discovery
Presents novel AI methodology for automating scientific validation, directly relevant to ML researchers working on reproducibility and scientific AI applications.
AI Summary
Researchers developed LLM agents that automate scientific reproducibility checks, successfully reproducing 77.4% of claims in medical studies and flagging 9.5% for expert review.
Excerpt
Reproducible and analytically robust research findings are essential to scientific progress, but evaluating these properties typically requires costly manual effort. We investigate whether LLM agents can help automate reproduction and robustness analysis, using a medical case study grounded in the Surveillance, Epidemiology, and End Results (SEER) database, a widely used U.S. cancer registry underlying tens of thousands of published studies. First, we introduce SEER-Repro, a benchmark of 20 publ
