Easy over Hard: A Simple Baseline for Test Failures Causes Prediction
The test failure causes analysis is critical since it determines the subsequent way of handling different types of bugs, which is the prerequisite to get the bugs properly analyzed and fixed. After a test case fails, software testers have to inspect the test execution logs line by line to identify its root cause. However, manual root cause determination is often tedious and time-consuming, which can cost 30-40% of the time needed to fix a problem. Therefore, there is a need for automatically predicting the test failure causes to lighten the burden of software testers. In this paper, we present a simple but hard-to-beat approach, named NCChecker (Naive Failure Cause Checker), to automatically identify the failure causes for failed test logs. Our approach can help developers efficiently identify the test failure causes, and flag the most probable log lines of indicating the root causes for investigation. Our approach has three main stages: log abstraction, lookup table construction, and failure causes prediction. We first perform log abstraction to parse the unstructured log messages into structured log events. NCChecker then automatically maintains and updates a lookup table via employing our heuristic rules, which record the matching score between different log events and test failure causes. When it comes to the failure cause prediction stage, for a newly generated failed test log, NCChecker can easily infer its failed reason by checking out the associated log events’ scores from the lookup table. We have developed a prototype and evaluated our tool on a real-world industrial dataset with more than 10K test logs. The extensive experiments show the promising performance of our model over a set of benchmarks. Moreover, our approach is highly efficient and memory-saving, and can successfully handle the data imbalance problem. Considering the effectiveness and simplicity of our approach, we recommend relevant practitioners to adopt our approach as a baseline to beat in the future.
Fri 19 JulDisplayed time zone: Brasilia, Distrito Federal, Brazil change
14:00 - 15:30 | Fault Diagnosis and Root Cause Analysis 2Research Papers / Industry Papers at Pitanga Chair(s): Xi Zheng Macquarie University | ||
14:00 18mTalk | Illuminating the Gray Zone: Non-Intrusive Gray Failure Localization in Server Operating Systems Industry Papers Shenglin Zhang Nankai University, Yongxin Zhao Nankai University, Xiao Xiong Nankai University, Yongqian Sun Nankai University, Xiaohui Nie CNIC, CAS, Jiacheng Zhang Nankai University, Fenglai Wang Huawei Technologies Ltd., Xian Zheng Huawei Technologies Ltd., Yuzhi Zhang Nankai University, Dan Pei Tsinghua University DOI File Attached | ||
14:18 18mTalk | Towards Better Graph Neural Network-based Fault Localization Through Enhanced Code Representation Research Papers Md Nakhla Rafi Concordia University, Dong Jae Kim Concordia University, An Ran Chen University of Alberta, Tse-Hsun (Peter) Chen Concordia University, Shaowei Wang Department of Computer Science, University of Manitoba, Canada | ||
14:36 18mTalk | Easy over Hard: A Simple Baseline for Test Failures Causes Prediction Industry Papers Zhipeng Gao Shanghai Institute for Advanced Study - Zhejiang University, Zhipeng Xue , Xing Hu Zhejiang University, Weiyi Shang University of Waterloo, Xin Xia Huawei Technologies |