A Miss Is as Good as A Mile: Metamorphic Testing for Deep Learning Operators
Deep learning (DL) is a critical tool for real-world applications, and comprehensive testing of DL models is vital to ensure their quality before deployment. However, recent studies have shown that even subtle deviations in DL operators can result in catastrophic consequences, underscoring the importance of rigorous testing of these components. Unlike testing other DL system components, operator analysis poses unique challenges due to complex inputs and uncertain outputs. The existing DL operator testing approach has limitations in terms of testing efficiency and error localization. In this paper, we propose Meta, a novel operator testing framework based on metamorphic testing that automatically tests and assists bug location based on metamorphic relations (MRs). Meta distinguishes itself in three key ways: (1) it considers both parameters and input tensors to detect operator errors, enabling it to identify both implementation and precision errors; (2) it uses MRs to guide the generation of more effective inputs (i.e., tensors and parameters) in less time; (3) it assists the precision error localization by tracing the error to the input level of the operator based on MR violations. We designed 21 metamorphic relations (MRs) for testing 10 widely used DL operators. To assess the effectiveness of our proposed Meta, we conducted experiments on 9 released versions of 5 popular DL libraries. Our results revealed that Meta successfully detected 32 bugs, including 14 new ones that were reported to the respective platforms and partially confirmed. Additionally, Meta demonstrated high efficiency, outperforming the baseline by detecting x~1.89 times more errors while only requiring x~ 0.5 times the cost of the baseline.
Thu 18 JulDisplayed time zone: Brasilia, Distrito Federal, Brazil change
16:00 - 18:00 | Testing 3Ideas, Visions and Reflections / Demonstrations / Research Papers / Journal First at Pitanga Chair(s): Qi Xin Wuhan University | ||
16:00 18mTalk | Search-based Software Testing Driven by Automatically Generated and Manually Defined Fitness Functions Journal First Federico Formica McMaster University, Tony Fan McMaster University, Claudio Menghi University of Bergamo; McMaster University | ||
16:18 9mTalk | Monitoring the Execution of 14K Tests: Methods Tend to Have One Path that Is Significantly More Executed Ideas, Visions and Reflections Andre Hora UFMG Pre-print Media Attached | ||
16:36 18mTalk | Finding and Understanding Defects in Static Analyzers by Constructing Automated Oracles Research Papers weigang he East China Normal University / University of Technology Sydney, Peng Di Ant Group, Mengli Ming East China Normal University, Chengyu Zhang ETH Zurich, Ting Su East China Normal University, Shijie Li Ant Group, Yulei Sui UNSW | ||
16:54 18mTalk | A Miss Is as Good as A Mile: Metamorphic Testing for Deep Learning Operators Research Papers Jinyin Chen Zhejiang University of Technology, Chengyu Jia Zhejiang University of Technology, Yunjie Yan Zhejiang University of Technology, Jie Ge Zhejiang University of Technology, haibin zheng Zhejiang University of Technology, Yao Cheng TÜV SÜD Asia Pacific Pte. Ltd. | ||
17:12 9mTalk | ExLi : An Inline-Test Generation Tool for Java Demonstrations Yu Liu University of Texas at Austin, Aditya Thimmaiah The University of Texas at Austin, Owolabi Legunsen Cornell University, Milos Gligoric The University of Texas at Austin | ||
17:21 9mTalk | ATheNA-S: a Testing Tool for Simulink Models Driven by Software Requirements and Domain Expertise Demonstrations Federico Formica McMaster University, Mohammad Mahdi Mahboob McMaster University, Mehrnoosh Askarpour McMaster University, Claudio Menghi University of Bergamo; McMaster University | ||
17:30 9mTalk | Test Polarity: Detecting Positive and Negative Tests Ideas, Visions and Reflections Andre Hora UFMG Pre-print Media Attached | ||
17:39 18mTalk | Java JIT Testing with Template Extraction Research Papers Zhiqiang Zang The University of Texas at Austin, Fu-Yao Yu The University of Texas at Austin, Aditya Thimmaiah The University of Texas at Austin, August Shi The University of Texas at Austin, Milos Gligoric The University of Texas at Austin DOI Pre-print |