Regression testing is a crucial part of software development. It checks that software changes do not break existing functionality. An important assumption of regression testing is that test outcomes are deterministic: an unmodified test is expected to either always pass or always fail for the same code under test. Unfortunately, in practice, some tests often called flaky tests—have non-deterministic outcomes. Such tests undermine the regression testing as they make it difficult to rely on test results. We present the first extensive study of flaky tests. We study in detail a total of 201 commits that likely fix flaky tests in 51 open-source projects. We classify the most common root causes of flaky tests, identify approaches that could manifest flaky behavior, and describe common strategies that developers use to fix flaky tests. We believe that our insights and implications can help guide future research on the important topic of (avoiding) flaky tests.

Fri 19 Jul

Displayed time zone: Brasilia, Distrito Federal, Brazil change

09:00 - 09:30
Test of Time Award Runner-Up Honorable MentionPlenary Events at Plenary Hall
09:00
30m
Keynote
An empirical analysis of flaky tests
Plenary Events
Qingzhou Luo University of Illinois at Urbana-Champaign, Farah Hariri University of Illinois at Urbana-Champaign, Lamyaa Eloussi , Darko Marinov University of Illinois at Urbana-Champaign
Link to publication DOI