TraceML: An Empirical Analysis of Human-Agent Planning in Machine Learning Development
Provides empirical evidence of human-AI planning differences in ML development workflows, with implications for agent design.
AI Summary
Researchers introduce TraceML, analyzing 4,465 Kaggle trajectories to show how human ML developers outperform AI agents through strategic pivoting and revisiting abandoned approaches.
Excerpt
Large language models write correct code for isolated problems but remain far weaker at autonomous machine-learning development, where an agent must revise data pipelines, models, and validation over hours of feedback, and on most competitions still finishes below strong human competitors. Outcome-based benchmarks record this gap but not its cause, because they grade the final submission and discard the development process behind it. We introduce TraceML, which pairs human and agent work on the