Costs and Benefits of Machine Learning Software Defect Prediction: Industrial Case Study
Context: Our research is set in the industrial context of Nokia 5G and the introduction of Machine Learning Software Defect Prediction (ML SDP) to the existing quality assurance process within the company. Objective: We aim to support or undermine the profitability of the proposed ML SDP solution designed to complement the system-level black-box testing at Nokia, as cost-effectiveness is the main success criterion for further feasibility studies leading to a potential commercial introduction. Method: To evaluate the expected cost-effectiveness, we utilize one of the available cost models for software defect prediction formulated by previous studies on the subject. Second, we calculate the standard Return on Investment (ROI) and Benefit-Cost Ratio (BCR) financial ratios to demonstrate the profitability of the developed approach based on real-world, business-driven examples. Third, we build an MS Excel-based tool to automate the evaluation of similar scenarios that other researchers and practitioners can use. Results: We considered different periods of operation and varying efficiency of predictions, depending on which of the two proposed scenarios were selected (lightweight or advanced). Performed ROI and BCR calculations have shown that the implemented ML SDP can have a positive monetary impact and be cost-effective in both scenarios. Conclusions: The cost of adopting new technology is rarely analyzed and discussed in the existing scientific literature, while it is vital for many software companies worldwide. Accordingly, we bridge emerging technology (machine learning software defect prediction) with a software engineering domain (5G system-level testing) and business considerations (cost efficiency) in an industrial environment of one of the leaders in 5G wireless technology.
Wed 17 JulDisplayed time zone: Brasilia, Distrito Federal, Brazil change
16:00 - 18:00 | Fault Diagnosis and Root Cause Analysis 1Demonstrations / Research Papers / Industry Papers at Sapoti Chair(s): Muhammad Ali Gulzar Virginia Tech | ||
16:00 18mTalk | A Quantitative and Qualitative Evaluation of LLM-based Explainable Fault Localization Research Papers Sungmin Kang Korea Advanced Institute of Science and Technology, Gabin An Korea Advanced Institute of Science and Technology, Shin Yoo Korea Advanced Institute of Science and Technology Pre-print | ||
16:18 18mTalk | BARO: Robust Root Cause Analysis for Microservices via Multivariate Bayesian Online Change Point Detection Research Papers Pre-print | ||
16:36 18mTalk | Fault Diagnosis for Test Alarms in Microservices Through Multi-source Data Industry Papers Shenglin Zhang Nankai University, Jun Zhu Nankai University, Bowen Hao Nankai University, Yongqian Sun Nankai University, Xiaohui Nie CNIC, CAS, Jingwen Zhu Nankai University, Xilin Liu Huawei Cloud, Xiaoqian Li Huawei Cloud, Yuchi Ma Huawei Cloud Computing Technologies CO., LTD., Dan Pei Tsinghua University | ||
16:54 18mTalk | Costs and Benefits of Machine Learning Software Defect Prediction: Industrial Case Study Industry Papers Szymon Stradowski Wroclaw University of Science and Technology & NOKIA, Lech Madeyski Wroclaw University of Science and Technology | ||
17:12 18mTalk | Chain-of-Event: Interpretable Root Cause Analysis for Microservices through Automatically Learning Weighted Event Causal Graph Industry Papers Zhenhe Yao Tsinghua University, Changhua Pei Computer Network Information Center at Chinese Academy of Sciences, Wenxiao Chen Tsinghua University, Hanzhang Wang Walmart Global Tech, Liangfei Su eBay, USA, Huai Jiang eBay, USA, Zhe Xie Tsinghua University, Xiaohui Nie CNIC, CAS, Dan Pei Tsinghua University | ||
17:30 18mTalk | ChangeRCA: Finding Root Causes from Software Changes in Large Online Systems Research Papers Guangba Yu Sun Yat-sen University, Pengfei Chen Sun Yat-sen University, Zilong He Sun Yat-sen University, Qiuyu Yan Tencent, Yu Luo Tencent, Fangyuan Li Tencent, Zibin Zheng Sun Yat-sen University DOI Pre-print | ||
17:48 9mTalk | MineCPP: Mining Bug Fix Pairs and Their Structures Demonstrations DOI Pre-print Media Attached |