An Empirical Study on ML DevOps Adoption Trends, Efforts, and Benefits Analysis

Abstract

Context: Machine Learning (ML), including Deep Learning(DL), based systems, have become ubiquitous in today’s solutions to many real-world problems. ML-based approaches are being applied to solve complex problems such as autonomous driving, recommendation systems, etc. Objective: To improve the quality and deliverability of ML-based applications, the software development community is adopting state-of-the-art DevOps practices within them. However, we currently lack knowledge about the DevOps adoption trends, maintenance efforts and benefits among MLbased projects, and this work attempts to remedy this knowledge-gap. Method: In this research work, we conducted a large-scale empirical analysis on 4031 ML projects, including 1116 ML Tools and 2915 ML Applied projects to quantify DevOps adoption, maintenance effort and benefits. To characterize the development behaviors, we performed configuration-scriptanalysis and commit-change-analysis on DevOps configuration files. To compare the characteristics of ML DevOps to those of traditional software projects, we performed the same analysis on 4076 non-ML projects. Results: Our analysis identified that ML projects, more specifically ML-Applied projects, have a slower, lower, and less efficient adoption of DevOps tools in general. DevOps configuration files in MLApplied projects tended to experience more frequent changes than ML-Tool projects and were less likely to occur in conjunction with build and bugfixes. It’s also evident that adopting DevOps in ML projects correlates with an increase in development productivity, code quality, and a decrease in bug resolution time, especially in ML-Applied projects which have the most to gain by adopting these tools. Conclusion: We identified the characteristics and improvement scopes of ML DevOps, such as the slower adoption of DevOps in certain ML projects, and the need for automatic configuration synchronization tools for these projects. We also identified the improvements the productivity of ML teams and projects associated with DevOps adoption, including better code quality, more frequent code sharing and integration and faster issue resolution.

Dhia Elhaq Rzig

Ph.D. Candidate in Computer and Information Science

My research interests include DevOps, Continuous Integration, Machine Learning, Testing and Virtual Reality.