Are mutation scores correlated with real fault detection?: a large scale empirical study on the relationship between mutants and real faults

摘要: Empirical validation of software testing studies is increasingly relying on mutants. This practice motivated by the strong correlation between mutant scores and real fault detection that reported in literature. In contrast, our study shows correlations are results confounding effects test suite size. particular, we investigate relation two independent variables, mutation score size, with one dependent variable (real) faults. We use data sets, CoreBench De-fects4J, large C Java programs faults provide evidence all weak when controlling for also found both variables significantly influence one, better fits, but overall relative low prediction power. By measuring capability top ranked, according to score, suites (opposed randomly selected same size), achieving higher improves detection. Taken together, suggest mutants good guidance improving suites, their weak.