A recent study by Model Evaluation and Threat Research (METR) has found that AI tools unexpectedly decreased software developer productivity, contrary to expectations. The study involved 16 experienced developers who used AI tools for half of their tasks and completed the other half without assistance. Results showed that task completion times increased by 19% when AI was used, compared to tasks done without AI.
The study's findings suggest that developers face significant challenges when working with AI tools, particularly when it comes to cleaning up AI-generated code to fit their projects. Some developers also lost time waiting for AI to generate results or writing prompts for chatbots. These findings highlight the complexities of integrating AI into existing workflows and the need for developers to adapt to new tools.
According to Anders Humlum, an assistant professor of economics, getting productivity gains from technology requires organizational adjustment, complementary investments, and worker skill improvements. This suggests that simply introducing AI tools into a workflow may not be enough to boost productivity, and that a more comprehensive approach is needed.
MIT economist Daron Acemoglu has also weighed in on the issue, suggesting that markets have overestimated productivity gains from AI. He argues that only 4.6% of tasks in the US economy will be made more efficient with AI, highlighting the limitations of the technology.
The study's findings have significant implications for the use of AI in software development. Rather than boosting productivity, AI may offer diminishing returns for skilled workers like experienced software developers. Forcing experienced workers to adopt AI tools may not be beneficial if they're already functioning well with existing methods. As the use of AI continues to grow, it's essential to understand its limitations and potential impact on productivity.
