DeFlaker: automatically detecting flaky tests

摘要: Developers often run tests to check that their latest changes a code repository did not break any previously working functionality. Ideally, new test failures would indicate regressions caused by the changes. However, some may be due but non-determinism in tests, popularly called flaky tests. The typical way detect is rerun failing repeatedly. Unfortunately, rerunning can costly and slow down development cycle.We present first extensive evaluation of propose technique, DeFlaker, detects if failure without with very low runtime overhead. DeFlaker monitors coverage marks as newly execute We deployed live, build process 96 Java projects on TravisCI, found 87 unknown 10 these projects. also ran experiments project histories, where detected 1, 874 from 4, 846 failures, false alarm rate (1.5%). had higher recall (95.5% vs. 23%) confirmed than Maven's default detector.