Agentic coding without the cloud: evaluating open-weight large language models on longitudinal data preparation tasks
Presents novel experimental framework and benchmarking for on-premise AI agents in data science workflows—essential for researchers dealing with sensitive data.
AI Summary
This research paper introduces an open-source framework for evaluating open-weight large language models on agentic coding tasks for longitudinal data preparation, specifically addressing governance-restricted research settings where sensitive data cannot leave local environments. The authors models across the consumer-grade deployment spectrum, analyzing their efficacy on 20 data preparation tasks using R code generation, finding that state-of-the-art 31-35B parameter models achieve up to 87.9% average task completion. The work demonstrates that locally deployable open-weight LLMs offer a viable path toward AI-assisted data preparation in research environments with strict data governance requirements.
Excerpt
Large language models (LLMs) and agents are now widely used tools in code development, with data typically sent to third-party cloud-based models. Their adoption in research using personal data is constrained by governance requirements that typically prohibit data transmission to external services. Locally deployable open-weight models offer an alternative since sensitive data never leave the local environment. We introduce an open-source framework for evaluating the efficacy of AI agents powere