On the estimation and validity of AI time horizons---a statistical look at the METR plot
Provides methodological improvements for AI capability benchmarking that researchers need for accurate model evaluation.
AI Summary
Researchers propose improved statistical methods for estimating AI time horizons using splines and item-response theory, revealing nonlinear relationships between human task completion time and AI difficulty.
Excerpt
METR's 50\% time horizon measures the human completion time of software tasks that an AI solves with 50\% probability, allowing AI capabilities to be expressed in interpretable units. On 228 tasks and 26 AIs, we recompute the time horizons using splines and item-response theory to relax the assumption that the AI difficulty of a task depends linearly on the log of human time. Our fitted spline can be interpreted as a function that \emph{converts} human time to AI difficulty; it is nearly flat in