The Replication Crisis in Software Engineering: Guidelines for the Scientific Community in dealing with Threats to Validity and their Mitigation Actions and the Impact of the 3R
Motivation: The Replication Crisis highlights challenges in reproducing results, emphasizing the importance of transparent, bias- controlled, and replicable empirical research. The Software Engineering (SE) community is committed to improving research quality by discussing and promoting best practices. Identifying validity threats and implementing control actions are key to ensuring replicability, enabling other researchers to reproduce results with confidence and accuracy. Objective: The objective of this study is to improve scientific practice in Software Engineering regarding Replication, Repetition, and Reproduction (3R) through the development and validation of guidelines with recommendations to reduce the risk of threats to validity in research by the treatment of these threats. Research Method: This research involves the following stages intending to develop our guidelines: first, we performed a Survey adapted from [ 1] to verify the SE replication-involved researchers’ perception regarding the Replication Crisis, challenges to carry out studies, good practices adopted by them, and about the definitions of the 3R. Second, we carried out a partial replication of the Systematic Mapping Studies (SMS) performed by Suassuna [ 8] and Bezerra [2] with a focus on articles published between 2013 and 2022 concerning Replication, Repetition, or Reproduction in SE. Third, we will use the Multiple Focus Group method to explore how threats impact empirical study results and conclusions. Finally, these findings will help us develop our recommendation guide. We will validate our guideline through a controlled experiment. Preliminary Results: The first stage reveals a high level of agreement with the definitions provided by the ACM on the 3R. However, disparities in opinions regarding the Replication Crisis were noted. Despite various factors contributing to the failure of the 3R, many researchers reported success in trying to reproduce or replicate other research. They emphasized the importance of addressing challenges and implementing practices to enhance the reliability and validity of scientific research. In the second stage, using the search string, we identified 12,053 articles. After applying inclusion and exclusion criteria, 340 potentially relevant studies remained for further analysis.
The Replication Crisis in Software Engineering: Guidelines for Mitigating Validity Threats in the Context of Replication, Repetition, and Reproduction (3R (FSE24__Doctoral_Symposium___Ivanildo (1).pdf) | 397KiB |
Mon 15 JulDisplayed time zone: Brasilia, Distrito Federal, Brazil change
16:00 - 18:00 | Empirical Aspects + Life After PhD Panel + ClosingDoctoral Symposium at Sapoti Chair(s): César França CESAR School, Raula Gaikovina Kula Nara Institute of Science and Technology, Ronnie de Souza Santos University of Calgary | ||
16:00 20mTalk | Open Science in Software Engineering: Unpacking the Black Box of Artifact Sharing Doctoral Symposium Ana Paula Vasconcelos Universidade Federal de Pernambuco / Universidade do Estado de Mato Grosso File Attached | ||
16:20 20mTalk | The Replication Crisis in Software Engineering: Guidelines for the Scientific Community in dealing with Threats to Validity and their Mitigation Actions and the Impact of the 3R Doctoral Symposium Ivanildo Azevedo Universidade Federal de Pernambuco File Attached | ||
16:40 50mPanel | Life After PhD Panel Doctoral Symposium César França CESAR School, Maria Teresa Baldassarre Department of Computer Science, University of Bari , Raula Gaikovina Kula Nara Institute of Science and Technology, Ronnie de Souza Santos University of Calgary | ||
17:30 10mDay closing | Closing Doctoral Symposium Maria Teresa Baldassarre Department of Computer Science, University of Bari , Wing Lam George Mason University |