Test Polarity: Detecting Positive and Negative Tests
Positive tests (aka, happy path tests) cover the expected behavior of the program, while negative tests (aka, unhappy path tests) check the unexpected behavior. Ideally, test suites should have both positive and negative tests to better protect against regressions. In practice, unfortunately, we cannot easily identify whether a test is positive or negative. A better understanding of whether a test suite is more positive or negative is fundamental to assessing the overall test suite capability in testing expected and unexpected behaviors. In this paper, we propose test polarity, an automated approach to detect positive and negative tests. Our approach runs/monitors the test suite and collects runtime data about the application execution to classify the test methods as positive or negative. In a first evaluation, test polarity correctly classified 117 tests as as positive or negative. Finally, we provide a preliminary empirical study to analyze the test polarity of 2,054 test methods from 12 real-world test suites of the Python Standard Library. We find that most of the analyzed test methods are negative (88%) and a minority is positive (12%). However, there is a large variation per project: while some libraries have an equivalent number of positive and negative tests, others have mostly negative ones.
Andre Hora is currently a professor in the Department of Computer Science at UFMG, Brazil. He received his PhD in Computer Science at the University of Lille, France. He was a Postdoctoral researcher at the ASERG group. He worked as a software engineer at Inria (Lille, France) and was research intern at Siemens (Erlangen, Germany).
Thu 18 JulDisplayed time zone: Brasilia, Distrito Federal, Brazil change
16:00 - 18:00 | Testing 3Ideas, Visions and Reflections / Demonstrations / Research Papers / Journal First at Pitanga Chair(s): Qi Xin Wuhan University | ||
16:00 18mTalk | Search-based Software Testing Driven by Automatically Generated and Manually Defined Fitness Functions Journal First Federico Formica McMaster University, Tony Fan McMaster University, Claudio Menghi University of Bergamo; McMaster University | ||
16:18 9mTalk | Monitoring the Execution of 14K Tests: Methods Tend to Have One Path that Is Significantly More Executed Ideas, Visions and Reflections Andre Hora UFMG Pre-print Media Attached | ||
16:36 18mTalk | Finding and Understanding Defects in Static Analyzers by Constructing Automated Oracles Research Papers weigang he East China Normal University / University of Technology Sydney, Peng Di Ant Group, Mengli Ming East China Normal University, Chengyu Zhang ETH Zurich, Ting Su East China Normal University, Shijie Li Ant Group, Yulei Sui UNSW | ||
16:54 18mTalk | A Miss Is as Good as A Mile: Metamorphic Testing for Deep Learning Operators Research Papers Jinyin Chen Zhejiang University of Technology, Chengyu Jia Zhejiang University of Technology, Yunjie Yan Zhejiang University of Technology, Jie Ge Zhejiang University of Technology, haibin zheng Zhejiang University of Technology, Yao Cheng TÜV SÜD Asia Pacific Pte. Ltd. | ||
17:12 9mTalk | ExLi : An Inline-Test Generation Tool for Java Demonstrations Yu Liu University of Texas at Austin, Aditya Thimmaiah The University of Texas at Austin, Owolabi Legunsen Cornell University, Milos Gligoric The University of Texas at Austin | ||
17:21 9mTalk | ATheNA-S: a Testing Tool for Simulink Models Driven by Software Requirements and Domain Expertise Demonstrations Federico Formica McMaster University, Mohammad Mahdi Mahboob McMaster University, Mehrnoosh Askarpour McMaster University, Claudio Menghi University of Bergamo; McMaster University | ||
17:30 9mTalk | Test Polarity: Detecting Positive and Negative Tests Ideas, Visions and Reflections Andre Hora UFMG Pre-print Media Attached | ||
17:39 18mTalk | Java JIT Testing with Template Extraction Research Papers Zhiqiang Zang The University of Texas at Austin, Fu-Yao Yu The University of Texas at Austin, Aditya Thimmaiah The University of Texas at Austin, August Shi The University of Texas at Austin, Milos Gligoric The University of Texas at Austin DOI Pre-print | ||
