How Well Industry-level Cause Bisection Works in Real-world - A Study on Linux Kernel
With the rapid development of automatic vulnerability detection, bug reporting becomes more frequent than before. However, bug fixing is still a laborious task. In bug-fixing progress, debugging needs much manual effort. To mitigate such efforts, various automatic analyses have been proposed to address the challenges of debugging, for example, locating bug-inducing changes. One of the representative approaches to automatically locate bug-inducing changes is cause bisection. It bisects a range of code change history and determines after which change the bug occurs. Although cause bisection has been applied in industrial testing systems for years, it still lacks a systematic understanding of it, which limits the further improvements of the current approaches. Thus, there is an urgent need to comprehensively evaluate the performance, limitations, and real-world impacts of the real-world cause bisection system to facilitate possible improvements.
In this paper, we take a popular industrial cause bisection system, i.e. the cause bisection of Syzbot, to perform an empirical study of real-world cause bisection practice. First, we construct a dataset consisting of 1,070 publicly disclosed bugs by Syzbot. Then, we investigate the overall performance ofcause bisection. Only one-third ofthe bisection results are correct. Moreover, we analyze the causes why cause bisection fails. More than 80% of failures are caused by unstable bug reproduction and unreliable bug triage. Furthermore, we discover that correct bisection results indeed facilitate bug-fixing, specifically, recommending the bug-fixing developer, indicating the bug-fixing location, and decreasing the bug-fixing time. Finally, to improve the performance of real-world cause bisection practice, we discuss possible improvements and future research directions.