AIDev dataset

Empirical Study of Quality and Security in AI-Generated Python Refactoring Pull Requests

Researchers conduct an empirical analysis of AI-agent-authored Python refactoring pull requests from the AIDev dataset, evaluating quality and security outcomes using PyQu, Pylint, and Bandit. Results show agentic commits improve a quality attribute in 22.5% of changes, while 24.17% of modified files introduce new Pylint issues and 4.7% introduce new Bandit security findings. Despite mixed quality outcomes, 73.5% of analyzed PRs are merged by developers. The study derives a taxonomy of 24 recurring change operations and argues for stronger tool-in-the-loop gating in AI-driven development workflows.