Evaluating Synthetic Bugs

摘要: Fuzz testing has been used to find bugs in programs since the 1990s, but despite decades of dedicated research, there is still no consensus on which fuzzing techniques work best. One reason for this paucity ground truth: real with known root causes and triggering inputs are difficult collect at a meaningful scale. Bug injection technologies that add synthetic into seem offer solution, differences finding these versus organic have not previously explored large Using over 80 years CPU time, we ran eight fuzzers across 20 targets from Rode0day bug-finding competition LAVA-M corpus. Experiments were standardized respect compute resources metrics gathered. These experiments show fuzzer performance as well impact various configuration options. For instance, it clear integrating symbolic execution mutational very effective using dictionaries improves performance. Other conclusions less clear-cut; example, one beat all others tests. It noteworthy found any (i.e., reported CVE), 50 such being available discovery A close analysis results revealed possible explanation: dramatic difference between where live "main path" discovered by fuzzers. We recent updates bug systems made more discover, they significantly easier than our target programs. Finally, study identifies flaws suggests number axes along should be improved.