Motivation: The Replication Crisis highlights challenges in reproducing results, emphasizing the importance of transparent, bias- controlled, and replicable empirical research. The Software Engineering (SE) community is committed to improving research quality by discussing and promoting best practices. Identifying validity threats and implementing control actions are key to ensuring replicability, enabling other researchers to reproduce results with confidence and accuracy. Objective: The objective of this study is to improve scientific practice in Software Engineering regarding Replication, Repetition, and Reproduction (3R) through the development and validation of guidelines with recommendations to reduce the risk of threats to validity in research by the treatment of these threats. Research Method: This research involves the following stages intending to develop our guidelines: first, we performed a Survey adapted from [ 1] to verify the SE replication-involved researchers’ perception regarding the Replication Crisis, challenges to carry out studies, good practices adopted by them, and about the definitions of the 3R. Second, we carried out a partial replication of the Systematic Mapping Studies (SMS) performed by Suassuna [ 8] and Bezerra [2] with a focus on articles published between 2013 and 2022 concerning Replication, Repetition, or Reproduction in SE. Third, we will use the Multiple Focus Group method to explore how threats impact empirical study results and conclusions. Finally, these findings will help us develop our recommendation guide. We will validate our guideline through a controlled experiment. Preliminary Results: The first stage reveals a high level of agreement with the definitions provided by the ACM on the 3R. However, disparities in opinions regarding the Replication Crisis were noted. Despite various factors contributing to the failure of the 3R, many researchers reported success in trying to reproduce or replicate other research. They emphasized the importance of addressing challenges and implementing practices to enhance the reliability and validity of scientific research. In the second stage, using the search string, we identified 12,053 articles. After applying inclusion and exclusion criteria, 340 potentially relevant studies remained for further analysis.

The Replication Crisis in Software Engineering: Guidelines for Mitigating Validity Threats in the Context of Replication, Repetition, and Reproduction (3R (FSE24__Doctoral_Symposium___Ivanildo (1).pdf)397KiB

Mon 15 Jul

Displayed time zone: Brasilia, Distrito Federal, Brazil change

16:00 - 18:00
Empirical Aspects + Life After PhD Panel + ClosingDoctoral Symposium at Sapoti
Chair(s): César França CESAR School, Raula Gaikovina Kula Nara Institute of Science and Technology, Ronnie de Souza Santos University of Calgary
16:00
20m
Talk
Open Science in Software Engineering: Unpacking the Black Box of Artifact Sharing
Doctoral Symposium
Ana Paula Vasconcelos Universidade Federal de Pernambuco / Universidade do Estado de Mato Grosso
File Attached
16:20
20m
Talk
The Replication Crisis in Software Engineering: Guidelines for the Scientific Community in dealing with Threats to Validity and their Mitigation Actions and the Impact of the 3R
Doctoral Symposium
Ivanildo Azevedo Universidade Federal de Pernambuco
File Attached
16:40
50m
Panel
Life After PhD Panel
Doctoral Symposium
César França CESAR School, Maria Teresa Baldassarre Department of Computer Science, University of Bari , Raula Gaikovina Kula Nara Institute of Science and Technology, Ronnie de Souza Santos University of Calgary
17:30
10m
Day closing
Closing
Doctoral Symposium
Maria Teresa Baldassarre Department of Computer Science, University of Bari , Wing Lam George Mason University