Background: Continuous Integration (CI) has become widely adopted to enable faster code change integration. Meanwhile, Machine Learning (ML) is being used by software applications for previously unsolvable real-world scenarios. ML projects employ development processes different from those of traditional software projects, but they too require multiple iterations in their development, and may benefit from CI. Aims: While there are many works covering CI within traditional software, none of them empirically explored the adoption of CI and its associated issues within ML projects. To address this knowledge gap, we performed an empirical analysis comparing CI adoption between ML and Non-ML projects. Method: We developed TraVanalyzer, the first Travis CI configuration ana- lyzer, to analyze the CI practices of ML projects, and developed a CI log analyzer to identify the different CI problems of ML projects. Results: We found that Travis CI is the most popular CI tool for ML projects, and that their CI adoption lags behind that of Non-ML projects, but that ML projects which adopted CI, used it for building, testing, code analysis, and automatic deployment more than Non- ML projects. Furthermore, while CI in ML projects is as likely to experience problems as CI in Non-ML projects, it has more varied reasons for build-breakage. The most frequent CI failures of ML projects are due to testing-related problems, similar to Non-ML and OSS CI failures. Conclusion: To the best of our knowledge, this is the first work that has analyzed ML projects’ CI usage, practices, and issues, and contextualized its results by comparing them with similar Non-ML projects. It provides findings for researchers and ML developers to identify possible improvement scopes for CI in ML projects

Context: Machine Learning (ML), including Deep Learning(DL), based systems, have become ubiquitous in today’s solutions to many real-world problems. ML-based approaches are being applied to solve complex problems such as autonomous driving, recommendation systems, etc. Objective: To improve the quality and deliverability of ML-based applications, the software development community is adopting state-of-the-art DevOps practices within them. However, we currently lack knowledge about the DevOps adoption trends, maintenance efforts and benefits among MLbased projects, and this work attempts to remedy this knowledge-gap. Method: In this research work, we conducted a large-scale empirical analysis on 4031 ML projects, including 1116 ML Tools and 2915 ML Applied projects to quantify DevOps adoption, maintenance effort and benefits. To characterize the development behaviors, we performed configuration-scriptanalysis and commit-change-analysis on DevOps configuration files. To compare the characteristics of ML DevOps to those of traditional software projects, we performed the same analysis on 4076 non-ML projects. Results: Our analysis identified that ML projects, more specifically ML-Applied projects, have a slower, lower, and less efficient adoption of DevOps tools in general. DevOps configuration files in MLApplied projects tended to experience more frequent changes than ML-Tool projects and were less likely to occur in conjunction with build and bugfixes. It’s also evident that adopting DevOps in ML projects correlates with an increase in development productivity, code quality, and a decrease in bug resolution time, especially in ML-Applied projects which have the most to gain by adopting these tools. Conclusion: We identified the characteristics and improvement scopes of ML DevOps, such as the slower adoption of DevOps in certain ML projects, and the need for automatic configuration synchronization tools for these projects. We also identified the improvements the productivity of ML teams and projects associated with DevOps adoption, including better code quality, more frequent code sharing and integration and faster issue resolution.