Finding and Understanding Defects in Static Analyzers by Constructing Automated Oracles
Static analyzers are playing crucial roles in helping find programming mistakes and security vulnerabilities. The correctness of their analysis results is crucial for the usability in practice. Otherwise, the potential defects in these analyzers (e.g., implementation errors, improper design choices) could affect the soundness (leading to false negatives) and precision (leading to false positives). However, finding the defects in off-the-shelf static analyzers is challenging because these analyzers usually lack clear and complete specifications, and the results of different analyzers may differ. To this end, this paper designs two novel types of automated oracles to find defects in static analyzers with randomly generated programs. The first oracle is constructed by using dynamic program executions and the second one leverages the inferred static analysis results. We applied these two oracles on three state-of-the-art static analyzers: Clang Static Analyzer (CSA), GCC Static Analyzer (GSA), and Pinpoint. We found 38 unique defects in these analyzers, 28 of which have been confirmed or fixed by the developers. We conducted a case study on these found defects followed by several insights and lessons learned for improving and better understanding static analyzers. We have made the artifacts publicly available at https://anonymous.4open.science/r/SA_Bugs-583F for replication and benefit the community.
Thu 18 JulDisplayed time zone: Brasilia, Distrito Federal, Brazil change
16:00 - 18:00 | Testing 3Ideas, Visions and Reflections / Demonstrations / Research Papers / Journal First at Pitanga Chair(s): Qi Xin Wuhan University | ||
16:00 18mTalk | Search-based Software Testing Driven by Automatically Generated and Manually Defined Fitness Functions Journal First Federico Formica McMaster University, Tony Fan McMaster University, Claudio Menghi University of Bergamo; McMaster University | ||
16:18 9mTalk | Monitoring the Execution of 14K Tests: Methods Tend to Have One Path that Is Significantly More Executed Ideas, Visions and Reflections Andre Hora UFMG Pre-print Media Attached | ||
16:36 18mTalk | Finding and Understanding Defects in Static Analyzers by Constructing Automated Oracles Research Papers weigang he East China Normal University / University of Technology Sydney, Peng Di Ant Group, Mengli Ming East China Normal University, Chengyu Zhang ETH Zurich, Ting Su East China Normal University, Shijie Li Ant Group, Yulei Sui UNSW | ||
16:54 18mTalk | A Miss Is as Good as A Mile: Metamorphic Testing for Deep Learning Operators Research Papers Jinyin Chen Zhejiang University of Technology, Chengyu Jia Zhejiang University of Technology, Yunjie Yan Zhejiang University of Technology, Jie Ge Zhejiang University of Technology, haibin zheng Zhejiang University of Technology, Yao Cheng TÜV SÜD Asia Pacific Pte. Ltd. | ||
17:12 9mTalk | ExLi : An Inline-Test Generation Tool for Java Demonstrations Yu Liu University of Texas at Austin, Aditya Thimmaiah The University of Texas at Austin, Owolabi Legunsen Cornell University, Milos Gligoric The University of Texas at Austin | ||
17:21 9mTalk | ATheNA-S: a Testing Tool for Simulink Models Driven by Software Requirements and Domain Expertise Demonstrations Federico Formica McMaster University, Mohammad Mahdi Mahboob McMaster University, Mehrnoosh Askarpour McMaster University, Claudio Menghi University of Bergamo; McMaster University | ||
17:30 9mTalk | Test Polarity: Detecting Positive and Negative Tests Ideas, Visions and Reflections Andre Hora UFMG Pre-print Media Attached | ||
17:39 18mTalk | Java JIT Testing with Template Extraction Research Papers Zhiqiang Zang The University of Texas at Austin, Fu-Yao Yu The University of Texas at Austin, Aditya Thimmaiah The University of Texas at Austin, August Shi The University of Texas at Austin, Milos Gligoric The University of Texas at Austin DOI Pre-print | ||